{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "92010cd5",
      "metadata": {},
      "source": [
        "# 2.5 Optimizing RAG applications to improve question-answer accuracy\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b0b18130",
      "metadata": {},
      "source": [
        "## 🚄 Preface  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b7042114",
      "metadata": {},
      "source": [
        "In the previous lessons, you have already identified some issues with the RAG chatbot through automated evaluations. However, optimizing prompts alone cannot fix problems caused by **inaccurate retrieval recall**, just as it would be difficult to provide the correct answer during an open-book exam if you were using the wrong reference material.\n",
        "\n",
        "In this section, you will gain a deeper understanding of the **RAG workflow** and work on improving the **accuracy of your RAG application’s question-answering**. This involves refining both the **retrieval** and **generation** phases to ensure that the model not only finds relevant information but also uses it effectively to produce accurate and reliable responses.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "37a39a58",
      "metadata": {},
      "source": [
        "## 🍁 Goals\n",
        "\n",
        "After completing this course, you will be able to:\n",
        "\n",
        "* Gain a deeper understanding of the implementation principles and technical details of RAG\n",
        "* Understand common issues with RAG applications and recommended solutions\n",
        "* Improve the performance of RAG applications through hands-on case studies\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5b0e80fa",
      "metadata": {},
      "source": [
        "## 1. Previous content recap\n",
        "\n",
        "In the previous chapter, you discovered that the Q&A bot was unable to adequately answer the question: \"Which department is Michael Johnson from?\" You can reproduce the issue using the following code:  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3f319db8",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Import the required dependency packages\n",
        "import os\n",
        "os.environ[\"TRANSFORMERS_VERBOSITY\"] = \"error\"\n",
        "from config.load_key import load_key\n",
        "import logging\n",
        "from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex, PromptTemplate\n",
        "from llama_index.embeddings.dashscope import DashScopeEmbedding, DashScopeTextEmbeddingModels\n",
        "from llama_index.llms.openai_like import OpenAILike\n",
        "from llama_index.core.node_parser import (\n",
        "    SentenceSplitter,\n",
        "    SemanticSplitterNodeParser,\n",
        "    SentenceWindowNodeParser,\n",
        "    MarkdownNodeParser,\n",
        "    TokenTextSplitter\n",
        ")\n",
        "from llama_index.core.postprocessor import MetadataReplacementPostProcessor\n",
        "from langchain_community.llms.tongyi import Tongyi\n",
        "from langchain_community.embeddings import DashScopeEmbeddings\n",
        "from datasets import Dataset\n",
        "from ragas import evaluate\n",
        "from ragas.metrics import context_recall, context_precision, answer_correctness\n",
        "from chatbot import rag\n",
        "from IPython.display import display"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "6525c224",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Set log level\n",
        "logging.basicConfig(level=logging.ERROR)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c1410463",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your configured API Key is: sk-4b*****\n"
          ]
        }
      ],
      "source": [
        "# Load API key\n",
        "load_key()\n",
        "# Do not print the API Key to logs in production environment to avoid leakage\n",
        "print(f'Your configured API Key is: {os.environ[\"DASHSCOPE_API_KEY\"][:5]+\"*\"*5}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b8651814",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Configure the Qwen LLM and text vector model\n",
        "Settings.llm = OpenAILike(\n",
        "    model=\"qwen-plus\",\n",
        "    api_base=\"https://dashscope.aliyuncs.com/compatible-mode/v1\",\n",
        "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),\n",
        "    is_chat_model=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "6a9d7c39",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "# Configure text vector model, set batch size and maximum input length\n",
        "Settings.embed_model = DashScopeEmbedding(\n",
        "    model_name=DashScopeTextEmbeddingModels.TEXT_EMBEDDING_V3,\n",
        "    embed_batch_size=6,\n",
        "    embed_input_length=8192\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "b970212a",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "# Define the question-answering function\n",
        "def ask(question, query_engine):\n",
        "    # Update the prompt template\n",
        "    rag.update_prompt_template(query_engine=query_engine)\n",
        "\n",
        "    # Output the question\n",
        "    print('=' * 50)  # Generate a dividing line using multiplication\n",
        "    print(f'🤔 Question: {question}')\n",
        "    print('=' * 50 + '\\n')  # Generate a dividing line using multiplication\n",
        "\n",
        "    # Get the answer\n",
        "    response = query_engine.query(question)\n",
        "\n",
        "    # Output the answer\n",
        "    print('🤖 Answer:')\n",
        "    if hasattr(response, 'print_response_stream') and callable(response.print_response_stream):\n",
        "        response.print_response_stream()\n",
        "    else:\n",
        "        print(str(response))\n",
        "\n",
        "    # Output reference documents\n",
        "    print('\\n' + '-' * 50)  # Generate a dividing line using multiplication\n",
        "    print('📚 Reference Documents:\\n')\n",
        "    for i, source_node in enumerate(response.source_nodes, start=1):\n",
        "        print(f'Document {i}:')\n",
        "        print(source_node)\n",
        "        print()\n",
        "\n",
        "    print('-' * 50)  # Generate a dividing line using multiplication\n",
        "\n",
        "    return response"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "e130e9dd",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "==================================================\n",
            "🤔 Question: Which department is Michael Johnson in?\n",
            "==================================================\n",
            "\n",
            "🤖 Answer:\n",
            "Michael Johnson is in the Technology department, as indicated by his position as System Administrator under Michael Chen, who is the CTO & Head of Technology. His office location is 456 Tech Hub #220.\n",
            "--------------------------------------------------\n",
            "📚 Reference Documents:\n",
            "\n",
            "Document 1:\n",
            "Node ID: 2a4e3738-f11d-488c-b5b7-4cc7a00d0aac\n",
            "Text: Employee Key Contact Information.md 2025-07-10 3 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-210Jennifer LeeMichael Chen456 Tech\n",
            "Hub #210Data Analyst(555) 234-\n",
            "5687jennifer.l@educompany.comPerformance metrics tracking\n",
            "EID-211Christopher LeeMichael Chen456 Tech Hub #211Security Analy...\n",
            "Score:  0.233\n",
            "\n",
            "\n",
            "Document 2:\n",
            "Node ID: 0a20a43c-9e42-4c34-b505-6ef4a79f061a\n",
            "Text: Employee Key Contact Information.md 2025-07-10 1 / 6Company\n",
            "Organizational Structure & Key Contact Information Corporate Mission\n",
            "Statement Our organization is dedicated to pioneering digital\n",
            "transformation in education through innovative learning solutions. As\n",
            "an edtech leader, we specialize in developing interactive online\n",
            "platforms that integr...\n",
            "Score:  0.232\n",
            "\n",
            "\n",
            "--------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "query_engine = rag.create_query_engine(rag.load_index())\n",
        "response = ask('Which department is Michael Johnson in?', query_engine)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "073b9153",
      "metadata": {},
      "source": [
        "You will find that the reason for this issue is that the correct reference information (document chunks) was not recalled during the retrieval phase. To improve this issue, you can apply a few simple strategies to preliminarily optimize the retrieval effect.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b1715f36",
      "metadata": {},
      "source": [
        "## 2. Initial optimization\n",
        "\n",
        "As mentioned in the introduction, you need to ensure that the LLM has access to the correct \"reference materials\" to provide accurate \"answers.\" Therefore, you can try increasing the number of \"reference materials\" retrieved or organizing the \"knowledge points\" in the reference materials into structured tables. You can start with the former:\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d3fd908d",
      "metadata": {},
      "source": [
        "### 2.1 Allowing LLMs to access more reference information\n",
        "\n",
        "Since the knowledge base contains information about Michael Johnson's employment history, you can expand the search scope and increase the probability of finding relevant information by recalling more document slices. In the previous code, only 2 document slices were retrieved. Now, you can increase the number of recalled slices to 5 and observe whether the retrieval performance improves."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d1cb6fc5",
      "metadata": {},
      "source": [
        "#### 2.1.1 Adjusting the code\n",
        "\n",
        "You can configure the following settings to allow the retrieval engine to recall the top 5 most relevant document slices.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "7ba86a10",
      "metadata": {},
      "outputs": [],
      "source": [
        "index = rag.load_index()\n",
        "query_engine = index.as_query_engine(\n",
        "    streaming=True,\n",
        "    # Retrieve 5 document slices at once, default is 2\n",
        "    similarity_top_k=5\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "624c4b9a",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "==================================================\n",
            "🤔 Question: Which department is Michael Johnson in?\n",
            "==================================================\n",
            "\n",
            "🤖 Answer:\n",
            "Michael Johnson appears in two different departments:\n",
            "\n",
            "1. **Technology Department**:\n",
            "   - **Position**: System Administrator\n",
            "   - **Office Location**: 456 Tech Hub #220\n",
            "   - **Email**: michael.j2@educompany.com\n",
            "\n",
            "2. **Course Development Department**:\n",
            "   - **Position**: QA Specialist\n",
            "   - **Office Location**: 456 Tech Hub #205\n",
            "   - **Email**: michael.j@educompany.com\n",
            "\n",
            "3. **Course Development Department**:\n",
            "   - **Position**: Technical Writer\n",
            "   - **Office Location**: 456 Tech Hub #209\n",
            "   - **Email**: michael.johnson@educompany.com\n",
            "\n",
            "If you need more specific information about which role he holds, please provide additional context or check the company intranet portal for the most current information.\n",
            "--------------------------------------------------\n",
            "📚 Reference Documents:\n",
            "\n",
            "Document 1:\n",
            "Node ID: 2a4e3738-f11d-488c-b5b7-4cc7a00d0aac\n",
            "Text: Employee Key Contact Information.md 2025-07-10 3 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-210Jennifer LeeMichael Chen456 Tech\n",
            "Hub #210Data Analyst(555) 234-\n",
            "5687jennifer.l@educompany.comPerformance metrics tracking\n",
            "EID-211Christopher LeeMichael Chen456 Tech Hub #211Security Analy...\n",
            "Score:  0.233\n",
            "\n",
            "\n",
            "Document 2:\n",
            "Node ID: 0a20a43c-9e42-4c34-b505-6ef4a79f061a\n",
            "Text: Employee Key Contact Information.md 2025-07-10 1 / 6Company\n",
            "Organizational Structure & Key Contact Information Corporate Mission\n",
            "Statement Our organization is dedicated to pioneering digital\n",
            "transformation in education through innovative learning solutions. As\n",
            "an edtech leader, we specialize in developing interactive online\n",
            "platforms that integr...\n",
            "Score:  0.232\n",
            "\n",
            "\n",
            "Document 3:\n",
            "Node ID: fb433d8d-e50f-42af-a47a-7f76bed915a3\n",
            "Text: Employee Key Contact Information.md 2025-07-10 5 / 6Employee\n",
            "IDNameSupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-602Kevin RobertsDavid Wilson321 People\n",
            "Place #405Recruitment Specialist(555) 456-\n",
            "7894kevin.r@educompany.comCandidate sourcing, interview coordination\n",
            "EID-603Sarah MitchellDavid Wilson321 Peo...\n",
            "Score:  0.225\n",
            "\n",
            "\n",
            "Document 4:\n",
            "Node ID: 7b8b5140-fdbc-4bbf-8b39-dbf030a33f18\n",
            "Text: Employee Key Contact Information.md 2025-07-10 2 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-102Maria GonzalezSarah Johnson123\n",
            "Innovation Dr #102Senior Instructional Designer(555) 123-\n",
            "4568maria.g@educompany.comCourse optimization, learner engagement\n",
            "analysis EID-103James WilsonSara...\n",
            "Score:  0.225\n",
            "\n",
            "\n",
            "Document 5:\n",
            "Node ID: 3d9d17f4-5368-4a6a-9a9a-50704ab008f8\n",
            "Text: Employee Key Contact Information.md 2025-07-10 6 / 6HR (training\n",
            "rollout) Document Version History VersionDate Description Author\n",
            "v1.02023-03-15Initial release Sarah Johnson v1.12023-04-01Added IT\n",
            "department details Robert Taylor v2.02023-06-10Complete restructuring\n",
            "David Wilson v2.12023-08-20Updated contact information Emily Davis\n",
            "v2.22023-10-0...\n",
            "Score:  0.223\n",
            "\n",
            "\n",
            "--------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "response = ask('Which department is Michael Johnson in?', query_engine)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5e1fa6d9",
      "metadata": {},
      "source": [
        "As you can see, after adjusting the number of recalls, your Q&A bot is now able to answer the question \"*Which department is Michael Johnson in?*\" This is because the recalled document slices already contain information about Michael Johnson and his department.\n",
        "\n",
        "However, simply increasing the number of recalled slices is not a good solution. Think about it—if this method could solve the problem, why not recall the entire knowledge base? That way, no information would be missed. But this would not only exceed the input length limit of LLMs, but also reduce the efficiency and accuracy of the model's responses due to excessive irrelevant information.\n",
        "\n",
        "In fact, there may be many colleagues named Michael Johnson in your company, which leads to another issue: when a user asks \"*Which department is Michael Johnson in?*\" , the system cannot determine which Michael Johnson the user is referring to. Simply increasing the number of recalls might retrieve information about multiple Michael Johnsons, but the system would still be unable to accurately decide which one's information to return. Therefore, we need to use other methods to further improve the RAG chatbot.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d852191e",
      "metadata": {},
      "source": [
        "#### 2.1.2 Evaluate improvement effectiveness\n",
        "\n",
        "To quantify the effectiveness of improvements in subsequent enhancements, you can continue to use Ragas from the previous chapter for evaluation. Suppose your company has three colleagues named Michael Johnson, who respectively work in the Teaching and Research, Course Development, and IT Department."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "1e36894d",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "# Define evaluation function\n",
        "def evaluate_result(question, response, ground_truth):\n",
        "    # Get the response content\n",
        "    if hasattr(response, 'response_txt'):\n",
        "        answer = response.response_txt\n",
        "    else:\n",
        "        answer = str(response)\n",
        "    # Get the retrieved context\n",
        "    context = [source_node.get_content() for source_node in response.source_nodes]\n",
        "\n",
        "    # Construct evaluation dataset\n",
        "    data_samples = {\n",
        "        'question': [question],\n",
        "        'answer': [answer],\n",
        "        'ground_truth': [ground_truth],\n",
        "        'contexts': [context],\n",
        "    }\n",
        "    dataset = Dataset.from_dict(data_samples)\n",
        "\n",
        "    # Evaluate using Ragas\n",
        "    score = evaluate(\n",
        "        dataset=dataset,\n",
        "        metrics=[answer_correctness, context_recall, context_precision],\n",
        "        llm=Tongyi(model_name=\"qwen-plus-0919\"),\n",
        "        embeddings=DashScopeEmbeddings(model=\"text-embedding-v3\")\n",
        "    )\n",
        "    return score.to_pandas()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "87734c7b",
      "metadata": {},
      "outputs": [],
      "source": [
        "question = 'Which department is Michael Johnson in?'\n",
        "ground_truth = '''There are three employees named Michael Johnson in the company:\n",
        "- Michael Johnson in the Teaching and Research Department: Position is Teaching and Research Specialist, email zhangwei@educompany.com.\n",
        "- Michael Johnson in the Course Development Department: Position is Course Development Specialist, email zhangwei01@educompany.com.\n",
        "- Michael Johnson in the IT Department: Position is IT Specialist, email zhangwei036@educompany.com.\n",
        "'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "182d8bea",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5b054b63c9494aa39f8768509c285ec2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson appears in two different depar...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Employee Key Contact Information.md 2025-07-1...</td>\n",
              "      <td>0.444665</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson appears in two different depar...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Employee Key Contact Information.md 2025-07-1...            0.444665   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.5  "
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluate_result(question=question, response=response, ground_truth=ground_truth)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fa99f324",
      "metadata": {},
      "source": [
        "As you can see, the current RAG system is still unable to operate efficiently. The retrieved document chunks contain irrelevant information, and the relevant information has not been fully recalled, resulting in an incorrect final answer. You must consider other improvement strategies."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "da8e8a0f",
      "metadata": {},
      "source": [
        "### 2.2 Provide more structured reference information\n",
        "\n",
        "In practical applications, the organizational structure of a document significantly impacts retrieval performance. Imagine this: the same information is placed either in a well-structured table or scattered throughout a block of plain text. Which one would be easier to locate and understand? Clearly, the former.\n",
        "\n",
        "The same applies to LLMs. When information originally presented in a table is converted into plain text, although no information is lost, its structure is diminished. This is akin to turning an organized drawer into a pile of scattered items—while everything is still there, it becomes less convenient to find things."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a4019ffe",
      "metadata": {},
      "source": [
        "#### 2.2.1 Rebuild the Index\n",
        "\n",
        "Markdown format is a great choice because it:\n",
        "* Has a clear structure and well-defined hierarchy\n",
        "* Simple syntax, making it easy to read and maintain\n",
        "* Is particularly suitable for organizing documents in RAG chatbot scenarios\n",
        "\n",
        "To validate the effectiveness of structured documents,  an optimized Markdown format file has been prepared for you. Next, you will:\n",
        "\n",
        "1. Add this Markdown file to the docs directory\n",
        "2. Rebuild the index\n",
        "3. Test the improvement in retrieval performance"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "e7f6e94c",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Copy the markdown formatted employee information document to the ./docs directory\n",
        "! mkdir -p ./docs/2_5\n",
        "! cp ./resources/2_4/Employee\\ Key\\ Contact\\ Information.md ./docs/2_5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "8ee5bbce",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "==================================================\n",
            "📂 Loading documents...\n",
            "==================================================\n",
            "\n",
            "✅ Document loading completed.\n",
            "\n",
            "==================================================\n",
            "🛠️ Rebuilding index...\n",
            "==================================================\n",
            "\n",
            "✅ Index rebuilding completed!\n",
            "==================================================\n"
          ]
        }
      ],
      "source": [
        "print('=' * 50)\n",
        "print('📂 Loading documents...')\n",
        "print('=' * 50 + '\\n')\n",
        "\n",
        "# Load documents\n",
        "documents = SimpleDirectoryReader('./docs/2_5').load_data()\n",
        "print(f'✅ Document loading completed.\\n')\n",
        "\n",
        "print('=' * 50)\n",
        "print('🛠️ Rebuilding index...')\n",
        "print('=' * 50 + '\\n')\n",
        "\n",
        "# Rebuild index\n",
        "index = VectorStoreIndex.from_documents(\n",
        "    documents\n",
        ")\n",
        "print('✅ Index rebuilding completed!')\n",
        "\n",
        "print('=' * 50)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "613d3642",
      "metadata": {},
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine(\n",
        "    streaming=True,\n",
        "    similarity_top_k=5\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "47495a4c",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "==================================================\n",
            "🤔 Question: Which department is Michael Johnson in?\n",
            "==================================================\n",
            "\n",
            "🤖 Answer:\n",
            "Michael Johnson is listed in the Course Development Department. There are two employees named Michael Johnson in this department:\n",
            "\n",
            "1. **Michael Johnson** - QA Specialist\n",
            "   - **Phone Number:** (555) 234-5682\n",
            "   - **Email Address:** michael.j@educompany.com\n",
            "   - **Key Responsibilities:** System validation testing\n",
            "\n",
            "2. **Michael Johnson** - Technical Writer\n",
            "   - **Phone Number:** (555) 234-5686\n",
            "   - **Email Address:** michael.johnson@educompany.com\n",
            "   - **Key Responsibilities:** Documentation creation\n",
            "\n",
            "If you need more specific information about either of these employees, please let me know!\n",
            "--------------------------------------------------\n",
            "📚 Reference Documents:\n",
            "\n",
            "Document 1:\n",
            "Node ID: 41c198cd-9cf2-46a6-a870-ffcc562ca85d\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document 2:\n",
            "Node ID: ca8404fa-9d04-4cf9-bd70-2b9bedd064cb\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document 3:\n",
            "Node ID: 82aa9d7f-79b6-4dcd-8413-9cb90de38f03\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document 4:\n",
            "Node ID: 125ce1e0-4012-463e-a629-8fdba7cb6eeb\n",
            "Text: Marketing & Outreach Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "Document 5:\n",
            "Node ID: 7c533904-96fa-429d-8046-b48f4b80fc71\n",
            "Text: Departmental Staff Directory\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "--------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "response = ask('Which department is Michael Johnson in?', query_engine)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "72f807bc",
      "metadata": {},
      "source": [
        "#### 2.2.2 Evaluate improvement effect\n",
        "\n",
        "You can see that your Q&A bot can accurately answer this question. You can run the Ragas evaluation again, and the evaluation data will also show that the answer accuracy has improved.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "b64aa328",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6062516cd09a44be94a8b96ca08a020a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is listed in the Course Develo...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Departmental Structure Overview\\n\\n| Departme...</td>\n",
              "      <td>0.366151</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is listed in the Course Develo...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Departmental Structure Overview\\n\\n| Departme...            0.366151   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.2  "
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "evaluate_result(question=question, response=response, ground_truth=ground_truth)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "247c76f9",
      "metadata": {},
      "source": [
        "## 3. Familiarize yourself with the RAG workflow\n",
        "\n",
        "So far, you have made some improvements to increase the accuracy of the Q&A for the RAG chatbot. However, in a real production environment, the problems you may encounter go far beyond this. Previously, you have already learned about some of the RAG workflow. Here, you can review the important steps to help you identify new areas for improvement:\n",
        "\n",
        "RAG is a technology that combines information retrieval and generative models, allowing it to leverage relevant information from an external knowledge base when generating answers. Its workflow can be divided into several key steps: parsing and slicing, vector storage, retrieval recall, and answer generation. You can refer back to the section \"Expanding the Knowledge Scope of the RAG chatbot\" for specific concepts.\n",
        "\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01J70UN61bQH46B7YUn_!!6000000003459-2-tps-1886-1100.png\" alt=\"RAG Working Principle\" width=\"1000px\">\n",
        "\n",
        "Next, we will focus on each step of RAG, and optimize its performance."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b2d2f244",
      "metadata": {},
      "source": [
        "## 4. Stages of RAG chatbot and improvement strategies\n",
        "\n",
        "### 4.1 Document preparation stage\n",
        "\n",
        "In traditional customer service systems, customer service personnel accumulate a knowledge base based on the questions raised by users, and share it with other staff for reference. This process is equally essential when building a RAG chatbot.\n",
        "\n",
        "* **Intent space**: We can map the needs behind user questions as points, which together form a user intent space.\n",
        "* **Knowledge space**: The knowledge points accumulated in the knowledge base documents constitute a knowledge space. These knowledge points can be a paragraph or a chapter.\n",
        "\n",
        "When we project the intent space and knowledge space together, we find that there are overlaps and differences between the two spaces. These areas correspond to our three subsequent optimization strategies:\n",
        "\n",
        "1. **Overlapping area**:\n",
        "    * This refers to parts where user questions can be answered based on the content of the knowledge base, forming the foundation of ensuring the effectiveness of the RAG chatbot.\n",
        "    * For these user intents, you can continuously improve the quality of responses through optimizing content quality, engineering, and algorithms.\n",
        "2. **Uncovered intent space**:\n",
        "    * Due to the lack of supporting content in the knowledge base, LLMs tend to generate hallucinations. For example, if the company has added a new \"Data Analysis Department,\" but there are no related documents in the knowledge base, no matter how much you improve the engineering algorithms, the RAG chatbot will not be able to accurately answer this question.\n",
        "    * What you need to do is proactively supplement the missing knowledge, and continually track changes in the user intent space.\n",
        "3. **Unused knowledge space**:\n",
        "    * Recalling irrelevant knowledge points may interfere with the LLM's responses.\n",
        "    * Therefore, you need to optimize the recall algorithm to avoid recalling unrelated content. Additionally, you should periodically check the knowledge base and remove irrelevant content.\n",
        "\n",
        "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01VA5rcf23b5eyUlYhW_!!6000000007273-2-tps-3770-1932.png\" alt=\"RAG Intent Space to Knowledge Space\" width=\"1000px\">\n",
        "\n",
        "Before attempting to optimize engineering or algorithms, you should prioritize building a mechanism that continuously collects user intents. By systematically gathering real user needs to enrich the knowledge base content and involving domain experts with deep understanding of user intents in the evaluation process, a closed-loop optimization process of \"data collection - knowledge update - expert validation\" is formed to ensure the effectiveness of the RAG chatbot.\n",
        "\n",
        "Once you have prepared these, you can further optimize various stages of the RAG chatbot."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "53d1a481",
      "metadata": {},
      "source": [
        "### 4.2 Document parsing and chunking phase\n",
        "\n",
        "The RAG application first parses the content of your document,  then slices the document content into chunks.\n",
        "\n",
        "If the document chunks that the LLM receives when answering questions lack key information, the response may be inaccurate. Similarly, if the  chunks contain too much irrelevant information (noise), it will also affect the quality of the response. In other words, either too little or too much information can impact the model's ability to generate effective responses.\n",
        "\n",
        "Therefore, during the document parsing and chunking phase, it is essential to ensure that the final chunks contain complete and relevant information without including excessive interfering content.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "186f3fb2",
      "metadata": {},
      "source": [
        "#### 4.2.1 Problem classification and improvement strategies\n",
        "\n",
        "During the document parsing and chunking phase, you may encounter the following issues:\n",
        "\n",
        "<table border=\"1\">\n",
        "  <thead>\n",
        "    <tr>\n",
        "      <th>Category</th>\n",
        "      <th>Subtype</th>\n",
        "      <th>Improvement Strategy</th>\n",
        "      <th>Scenario Example</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td rowspan=\"3\">Document Parsing</td>\n",
        "      <td>Non-uniform document types. Some formats are not supported for parsing; for example, SimpleDirectoryLoader used earlier does not support Keynote files.</em></td>\n",
        "      <td>Develop a parser for the corresponding format or convert the document format.</td>\n",
        "      <td>A company uses a large number of Keynote files to store employee information, but the existing parser does not support the Keynote format. A Keynote parser can be developed, or the files can be converted into a supported format (such as  PDF).</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Within the already supported document formats, there is some special content, such as embedded tables, images, and videos.</em></td>\n",
        "      <td>Improve the document parser.</td>\n",
        "      <td>A document contains many tables and images, and the current parser cannot correctly extract information from the tables. The parser can be improved to handle tables and images.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td rowspan=\"4\">Document Chunking</td>\n",
        "      <td>The document contains much content with similar themes. For example, in a work manual, each stage, including requirements analysis, development, and release, has precautions and operational guidance.</em></td>\n",
        "      <td>Expand document titles and subtitles: \"Precautions\" => \"Requirements Analysis > Precautions\"; create document metadata (tagging).</td>\n",
        "      <td>A document contains precautions for multiple stages. When a user asks, \"What are the precautions for requirements analysis?\" the system returns precautions for all stages. Titles can be expanded and tagging can be used to distinguish content across different stages.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Document chunks are too long, introducing excessive noise.</td>\n",
        "      <td>Reduce chunk length, or develop more suitable chunking strategies based on specific business needs.</td>\n",
        "      <td>A document's chunks are too long and contain multiple unrelated topics, resulting in irrelevant information being returned during retrieval. Chunk length can be reduced to ensure that each chunk contains only one topic.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Document chunks are too short, truncating useful information.</td>\n",
        "      <td>Increase chunk length, or develop more suitable chunking strategies based on specific business needs.</td>\n",
        "      <td>Each chunk in a document contains only one sentence, making it impossible to retrieve complete context during search. Chunk length can be increased to ensure that each chunk contains complete context.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "34175364",
      "metadata": {},
      "source": [
        "#### 4.2.2 Parsing PDF files using Model Studio\n",
        "\n",
        "In the previous sections of this course, we provided a Markdown document converted from a PDF so that you could quickly see the effects of format conversion. However, in real-world work scenarios, writing code to properly convert PDFs into Markdown is not an easy task.\n",
        "\n",
        "In practical work, you can also use DashScopeParse provided by Model Studio to parse files in formats such as PDF and Word. Behind DashScopeParse lies Alibaba Cloud's [Document Intelligence](https://www.aliyun.com/product/ai/docmind) service, which uses image recognition  to recognize images within documents, and optical character recognition (OCR) to extract structured text information from files in formats like PDF and Word.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "7d53df69",
      "metadata": {},
      "outputs": [],
      "source": [
        "from llama_index.readers.dashscope.utils import ResultType\n",
        "from llama_index.readers.dashscope.base import DashScopeParse\n",
        "import os\n",
        "import json\n",
        "import nest_asyncio"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "70bce4c2",
      "metadata": {},
      "outputs": [],
      "source": [
        "nest_asyncio.apply()\n",
        "# Use environment variables\n",
        "os.environ['DASHSCOPE_API_KEY'] = os.getenv('DASHSCOPE_API_KEY')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "id": "a69e566a",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "# Create a silent logger to replace the original logger\n",
        "silent_logger = logging.getLogger(__name__)\n",
        "# Set the log level to ERROR to avoid outputting irrelevant information. If you need to view more detailed log information, set it to INFO\n",
        "silent_logger.setLevel(logging.ERROR)\n",
        "\n",
        "class SilentDashScopeParse(DashScopeParse):\n",
        "    def __init__(self, *args, **kwargs):\n",
        "        # Replace the logger in all related modules\n",
        "        import llama_index.readers.dashscope.base as base_module\n",
        "        import llama_index.readers.dashscope.domain.lease_domains as lease_domains_module\n",
        "        import llama_index.readers.dashscope.utils as utils_module\n",
        "\n",
        "        base_module.logger = silent_logger\n",
        "        lease_domains_module.logger = silent_logger\n",
        "        utils_module.logger = silent_logger\n",
        "\n",
        "        # Call the parent class initialization\n",
        "        super().__init__(*args, **kwargs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "id": "779386c7",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Parsing files: 100%|██████████| 1/1 [00:09<00:00,  9.04s/it]"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Parsed Markdown text:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "Employee Key Contact Information.md\n",
            "\n",
            "2025-07-10\n",
            "\n",
            "[empty]\n",
            "1/ 6\n",
            "\n",
            "Company Organizational Structure & Key Contact Information\n",
            "\n",
            "Corporate Mission Statement\n",
            "\n",
            "Our organization is dedicated to pioneering digital transformation in education through innovative learning solutions. As an edtech leader, we specialize in developing interactive online platforms that integrate cutting-edge educational research with emerging technologies. Our strategic vision focuses on three core pillars:\n",
            "\n",
            "1. Personalized Learning: Creating adaptive platforms that cater to diverse learning styles and paces\n",
            "\n",
            "2. Educational Equity: Ensuring accessible, high-quality resources for all demographics\n",
            "\n",
            "3. Lifelong Learning: Developing continuous education pathways that support career advancement\n",
            "\n",
            "We maintain rigorous quality standards through our comprehensive evaluation framework, which includes:\n",
            "\n",
            "·Learner engagement metrics\n",
            "\n",
            "·Pedagogical effectiveness assessments\n",
            "\n",
            "·Technological innovation benchmarks\n",
            "\n",
            "Departmental Structure Overview\n",
            "\n",
            "Interdepartmental\n",
            "\n",
            "Department Name       Core Responsibilities\n",
            "\n",
            "|  | |Partnerships|\n",
            "| ---|---|---|\n",
            "| Instructional Design|Curriculum development, pedagogical research, learning outcomes<br>analysis|Course Development, Evaluation|\n",
            "| Course Development|Technical implementation of educational content, platform integration|IT, Marketing|\n",
            "| Textbook Production|Content curation, exercise design, multimedia resource development|Instructional Design|\n",
            "| Quality Assurance|Content validation, compliance monitoring, learner feedback analysis|All departments|\n",
            "| Marketing & Outreach|Brand positioning, market research, partnership development|Instructional Design|\n",
            "| Human Resources|Talent acquisition, professional development, employee engagement|All departments|\n",
            "| IT Infrastructure|System maintenance, cybersecurity protocols, technical support|All departments|\n",
            "\n",
            "Performance\n",
            "\n",
            "KPl tracking,360-degree evaluations, compensation alignment          HR, Finance\n",
            "\n",
            "Management\n",
            "\n",
            "Executive Leadership Team\n",
            "\n",
            "| Name|Position|Office Location|Contact Info|\n",
            "| ---|---|---|---|\n",
            "| Sarah Johnson|Chief Learning Officer (CLO)|123Innovation Dr #101|(555) 123-4567|\n",
            "| Michael Chen|CTO & Head of Technology|456 Tech Hub #202|(555)234-5678|\n",
            "| Emily Davis|Director of Marketing|789 Market Ave #303|(555)345-6789|\n",
            "| David Wilson|VP of Human Resources|321 People Place#404|(555)456-7890|\n",
            "| Robert Taylor|Director of IT Infrastructure|654 Tech Tower #505|(555) 567-8901|\n",
            "\n",
            "Departmental Staff Directory\n",
            "\n",
            "Instructional Design Department\n",
            "\n",
            "| Employee|Name|Supervisor|Office|Position Title|Phone|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| ID|Name|Supervisor|Location|Position Title|Number|Email Address|Responsibilities|\n",
            "| EID-101| | | | | |john.s@educompany.com|Curriculum|\n",
            "| EID-101|John|Sarah|123|Lead|(555)|john.s@educompany.com|architecture,|\n",
            "| EID-101|John|Sarah|Innovation|Instructional|123-|john.s@educompany.com|pedagogical|\n",
            "| EID-101|Smith|Johnson|Dr#101|Designer|4567|john.s@educompany.com|strategy, learning|\n",
            "| EID-101| | | | | | |analytics|\n",
            "\n",
            "[empty]\n",
            "Employee Key Contact Information.md\n",
            "\n",
            "2025-07-10\n",
            "\n",
            "[empty]\n",
            "2/ 6\n",
            "\n",
            "| Employee<br>ID|Name|Supervisor|Office|Position Title|Phone|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| Employee<br>ID|Name|Supervisor|Location|Position Title|Number|Email Address|Responsibilities|\n",
            "| EID-102|Maria|Sarah|123|Senior|(555)| |Course optimization,|\n",
            "| EID-102|Maria|Sarah|Innovation|Instructional|123-|maria.g@educompany.com|learner engagement|\n",
            "| EID-102|Gonzalez|Johnson|Dr#102|Designer|4568| |analysis|\n",
            "| EID-102| | | | | | |Content|\n",
            "| EID-103| | |123| |(555)| |Content|\n",
            "| EID-103|James|Sarah|123|Instructional|(555)| |development,|\n",
            "| EID-103|James|Sarah|Innovation<br>Dr#103|Instructional|123-<br>4569|james.w@educompany.com|development,|\n",
            "| EID-103|Wilson|Johnson|Innovation<br>Dr#103|Designer|123-<br>4569|james.w@educompany.com|multimedia|\n",
            "\n",
            "integration\n",
            "\n",
            "Course Development Department (Note: Scroll required for full visibility)\n",
            "\n",
            "| Employee|Name|Supervisor|Office|Position|Phone|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| ID|Name|Supervisor|Location|Title|Number|Email Address|Responsibilities|\n",
            "| EID-201|Karen Lee| |456| |(555)|karen.l@educompany.com|Platform<br>architecture, APl|\n",
            "| EID-201|Karen Lee|Michael|Tech|Lead Course|(555)|karen.l@educompany.com|Platform<br>architecture, APl|\n",
            "| EID-201|Karen Lee|Michael|Tech|Lead Course|234-|karen.l@educompany.com|Platform<br>architecture, APl|\n",
            "| EID-201|Karen Lee|Chen|Hub|Developer| |karen.l@educompany.com|integration|\n",
            "| EID-201|Karen Lee| |#201| |5678|karen.l@educompany.com| |\n",
            "|  | | |456|Senior|(555)| |LMS|\n",
            "| EID-202|Thomas|Michael|Tech|Course|234-|thomas.n@educompany.com|development,|\n",
            "|  |Nguyen|Chen|Hub|Developer|5679| |UXoptimization|\n",
            "|  | | |#202| | | | |\n",
            "| EID-203| | |456| |(555)|olivia.m@educompany.com|Content|\n",
            "| EID-203|Olivia|Michael|Tech|Course|234-|olivia.m@educompany.com|migration,|\n",
            "| EID-203|Martin|Chen|Hub|Developer|5680|olivia.m@educompany.com|system testing|\n",
            "| EID-203| | |#203| | |olivia.m@educompany.com| |\n",
            "|  | | |456|Junior|(555)| |Entry-level|\n",
            "| EID-204|John Smith|Michael|Tech|Course|234-|john.smith1@educompany.com|development|\n",
            "|  | |Chen|Hub|Developer|5681| |tasks|\n",
            "|  | | |#204| | | | |\n",
            "| EID-205| | |456| |(555)| |System|\n",
            "| EID-205|Michael|Michael|Tech|QA Specialist|234-|michael.j@educompany.com|validation|\n",
            "| EID-205|Johnson|Chen|Hub<br>#205| |5682| |testing|\n",
            "|  | | |456| |(555)| | |\n",
            "|  | |Michael|Tech|Content|(555)| |Resource|\n",
            "| EID-206|Sarah Miller|Michael|Tech|Content|234-<br>5683|sarah.m@educompany.com|Resource|\n",
            "| EID-206|Sarah Miller|Chen|Hub|Integrator|234-<br>5683|sarah.m@educompany.com|coordination|\n",
            "|  | | |#206| | | | |\n",
            "|  | | |456| |(555)| | |\n",
            "| EID-207|David Kim|Michael|Tech|UX Designer|234-<br>5684|david.k@educompany.com|Interface|\n",
            "| EID-207|David Kim|Chen|Hub|UX Designer|234-<br>5684|david.k@educompany.com|optimization|\n",
            "|  | | |#207| |234-<br>5684| | |\n",
            "|  | | |456| |(555)| | |\n",
            "| EID-208|Emily Davis|Michael|Tech|API|234-<br>5685|emily.d@educompany.com|Backend system|\n",
            "| EID-208|Emily Davis|Chen|Hub|Developer|234-<br>5685|emily.d@educompany.com|integration|\n",
            "|  | | |#208| |234-<br>5685| | |\n",
            "|  | | |456| |(555)| | |\n",
            "| EID-209|Michael|Michael|Tech|Technical|234-<br>5686|michael.johnson@educompany.com|Documentation|\n",
            "| EID-209|Johnson|Chen|Hub|Writer|234-<br>5686|michael.johnson@educompany.com|creation|\n",
            "|  | | |#209| | | | |\n",
            "| Employee|Name|Supervisor|Office|Position|Phone|Email Address|Key|\n",
            "| ID|Name|Supervisor|Location|Title|Number|Email Address|Responsibilities|\n",
            "| EID-210| | |456|Data Analyst|(555)|jennifer.l@educompany.com| |\n",
            "| EID-210|Jennifer|Michael|Tech|Data Analyst|(555)|jennifer.l@educompany.com|Performance|\n",
            "| EID-210|Jennifer|Michael|Tech|Data Analyst|234-|jennifer.l@educompany.com|Performance|\n",
            "| EID-210|Lee|Chen|Hub<br>#210|Data Analyst|5687|jennifer.l@educompany.com|metrics tracking|\n",
            "| EID-211| | |456| |(555)|christopher.l@educompany.com|System|\n",
            "| EID-211|Christopher|Michael|Tech|Security|234-|christopher.l@educompany.com|vulnerability|\n",
            "| EID-211|Lee|Chen|Hub<br>#211|Analyst|5688|christopher.l@educompany.com|assessments|\n",
            "|  | | |456| |(555)| | |\n",
            "|  | | | | |(555)|olivia.t@educompany.com| |\n",
            "| EID-212|Olivia|Michael|Tech|Project|234-|olivia.t@educompany.com|Timeline|\n",
            "| EID-212|Taylor|Chen|Hub|Coordinator|234-|olivia.t@educompany.com|management|\n",
            "|  |Taylor|Chen|Hub|Coordinator|5689|olivia.t@educompany.com|management|\n",
            "|  | | |#212| |5689| | |\n",
            "| EID-213| | |456| |(555)| | |\n",
            "| EID-213|Daniel|Michael|Tech|Frontend|234-|daniel.s@educompany.com|UI|\n",
            "| EID-213|Smith|Chen|Hub|Developer|5690| |implementation|\n",
            "| EID-213| | |#213| |5690| | |\n",
            "|  | | |456| |(555)| | |\n",
            "|  | |Michael|Tech|Mobile App| | |Cross-platform|\n",
            "| EID-214|Rachel Kim|Chen|Hub|Developer|234-<br>5691|rachel.k@educompany.com|development|\n",
            "|  | | |#214| | | | |\n",
            "|  | | |456| |(555)| | |\n",
            "| EID-215|Thomas|Michael|Tech|Cloud|234-|thomas.nguyen@educompany.com|Infrastructure|\n",
            "| EID-215|Nguyen|Chen|Hub|Engineer|234-|thomas.nguyen@educompany.com|management|\n",
            "|  |Nguyen|Chen|Hub|Engineer|5692|thomas.nguyen@educompany.com|management|\n",
            "|  | | |#215| |5692| | |\n",
            "|  | | |456| |(555)| |Continuous|\n",
            "| EID-216|Sophia Li|Michael|Tech|DevOps|234-|sophia.l@educompany.com|integration|\n",
            "|  | |Chen|Hub|Specialist|5693| |pipelines|\n",
            "|  | | |#216| | | | |\n",
            "|  | | |456| | | | |\n",
            "| EID-217| | | | |(555)| |Data|\n",
            "| EID-217|William|Michael|Tech|Database|234-|william.b@educompany.com|architecture|\n",
            "| EID-217|Brown|Chen|Hub|Administrator|5694| |design|\n",
            "|  | | |#217| | | | |\n",
            "|  | | |456| |(555)| | |\n",
            "| EID-218|Emma|Michael|Tech|Technical|234-|emma.w@educompany.com|User assistance|\n",
            "|  |Wilson|Chen|Hub|Support|5695| | |\n",
            "|  | | |#218| | | | |\n",
            "|  | | |456| |(555)| | |\n",
            "| EID-219|John Smith|Michael|Tech|Junior QA|234-<br>5696|john.smith2@educompany.com|Basic testing|\n",
            "| EID-219|John Smith|Chen|Hub|Tester|234-<br>5696|john.smith2@educompany.com|procedures|\n",
            "|  | | |#219| |234-<br>5696| | |\n",
            "| EID-220| | |456| | | | |\n",
            "| EID-220| | | | |(555)| | |\n",
            "| EID-220|Michael|Michael|Tech|System|234-| |Server|\n",
            "| EID-220|Johnson|Chen|Hub|Administrator|5697|michael.j2@educompany.com|maintenance|\n",
            "| EID-220| | |#220| | | | |\n",
            "| Textbook Production Department|Textbook Production Department|Textbook Production Department|Textbook Production Department|Textbook Production Department|Textbook Production Department|Textbook Production Department|Textbook Production Department|\n",
            "\n",
            "Employee Key Contact Information.md\n",
            "\n",
            "2025-07-10\n",
            "\n",
            "Employee                         Office        Position      Phone\n",
            "\n",
            "Name   Supervisor                                         Email Address             Key Responsibilities\n",
            "\n",
            "ID                                Location     Title          Number\n",
            "\n",
            "[empty]\n",
            "Employee Key Contact Information.md\n",
            "\n",
            "2025-07-10\n",
            "\n",
            "[empty]\n",
            "[empty]\n",
            "Employee                         Office        Position      Phone\n",
            "\n",
            "Name   Supervisor                                         Email Address             Key Responsibilities\n",
            "\n",
            "ID                               Location     Title         Number\n",
            "\n",
            "Lead          (555)\n",
            "\n",
            "Daniel    Sarah       789 Market                                                       Content curation,\n",
            "\n",
            "| EID-301| | | | | |daniel.k@educompany.com| |\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| EID-301| | | |Textbook|345-|daniel.k@educompany.com| |\n",
            "| EID-301|Kim|Johnson|Ave #301|Editor|6789|daniel.k@educompany.com|editorial standards|\n",
            "|  | | | |Senior|(555)| |Exercise|\n",
            "| EID-302|Sophia|Sarah|789 Market|Textbook|345-<br>6790|sophia.p@educompany.com|development,|\n",
            "| EID-302|Patel|Johnson|Ave#302|Textbook|345-<br>6790|sophia.p@educompany.com|multimedia asset<br>coordination|\n",
            "|  |Patel|Johnson|Ave#302|Editor|345-<br>6790| |multimedia asset<br>coordination|\n",
            "| EID-303| | | | |(555)|william.h@educompany.com| |\n",
            "| EID-303|William|Sarah|789 Market|Textbook|345-|william.h@educompany.com|Quality assurance,|\n",
            "| EID-303|Harris|Johnson|Ave#303|Editor|6791|william.h@educompany.com|compliance checks|\n",
            "\n",
            "Quality Assurance Department\n",
            "\n",
            "| Employee<br>ID|Name|Supervisor|Office|Position|Phone|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| Employee<br>ID|Name|Supervisor|Location|Title|Number|Email Address|Responsibilities|\n",
            "| EID-401|Emily White| | | |(555)|emily.w@educompany.com|Testing|\n",
            "| EID-401|Emily White|David|321 People|QA|(555)|emily.w@educompany.com|protocols,|\n",
            "| EID-401|Emily White|David|321 People|QA|456-<br>7890|emily.w@educompany.com|protocols,|\n",
            "| EID-401|Emily White|Wilson|Place#401|Manager|456-<br>7890|emily.w@educompany.com|certification|\n",
            "| EID-401|Emily White| | | | |emily.w@educompany.com|audits|\n",
            "| EID-402| | | | | |christopher.l@educompany.com|Learner|\n",
            "| EID-402|Christopher|David|321People|QA|(555)|christopher.l@educompany.com|feedback|\n",
            "| EID-402|Christopher|David|321People|QA|456-|christopher.l@educompany.com|analysis,|\n",
            "| EID-402|Lee|Wilson|Place#402|Analyst|7891|christopher.l@educompany.com|compliance|\n",
            "| EID-402| | | | | |christopher.l@educompany.com|monitoring|\n",
            "| EID-403| | | | | |rachel.a@educompany.com|monitoring|\n",
            "| EID-403| | | | |(555)|rachel.a@educompany.com|Content|\n",
            "| EID-403|Rachel|David|321 People|QA|456-|rachel.a@educompany.com|validation,|\n",
            "| EID-403|Adams|Wilson|Place#403|Specialist|7892|rachel.a@educompany.com|system testing|\n",
            "\n",
            "Marketing & Outreach Department\n",
            "\n",
            "Employee\n",
            "\n",
            "Office                       Phone\n",
            "\n",
            "Key\n",
            "\n",
            "Name      Supervisor              Position Title             Email Address\n",
            "\n",
            "ID                                 Location                    Number                              Responsibilities\n",
            "\n",
            "| 654 Tech(555)Brand strategy,\tMarketing\tEID-501Lisa ChenEmily DavisTower567-lisa.c@educompany.compartnership\tDirector\t#5018901development|\n",
            "| ---|\n",
            "| 654 TechDigital(555)Social media\tMarkEID-502                Emily Davis   Tower       Marketing567-mark.t@educompany.comcampaigns, SEO\tThompson#502Specialist8902optimization|\n",
            "\n",
            "Educational\n",
            "\n",
            "654 Tech    Content         (555)                                  content\n",
            "\n",
            "Amanda\n",
            "\n",
            "EID-503                Emily Davis   Tower                          567-\n",
            "\n",
            "Marketing               amanda.w@educompany.com  curation.\n",
            "\n",
            "Walker\n",
            "\n",
            "#503        Specialist       8903                                 thought\n",
            "\n",
            "leadership\n",
            "\n",
            "Human Resources Department\n",
            "\n",
            "| Employee<br>ID|Name|Supervisor|Office|Position Title|Phone|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| Employee<br>ID|Name|Supervisor|Location|Position Title|Number|Email Address|Responsibilities|\n",
            "| EID-601| | | |HR Director|(555)|jennifer.l@educompany.com|Talent acquisition,|\n",
            "| EID-601|Jennifer|David|321 People|HR Director|456-|jennifer.l@educompany.com|employee|\n",
            "| EID-601|Lopez|Wilson|Place #404|HR Director|7893|jennifer.l@educompany.com|relations|\n",
            "\n",
            "Employee Key Contact Information.md\n",
            "\n",
            "2025-07-10\n",
            "\n",
            "5/ 6\n",
            "\n",
            "Employee\n",
            "\n",
            "Office                             Phone                               Key\n",
            "\n",
            "Name    Supervisor                 Position Title                 Email Address\n",
            "\n",
            "ID                                     Location                           Number                                 Responsibilities\n",
            "\n",
            "[empty]\n",
            "Candidate\n",
            "\n",
            "(555)\n",
            "\n",
            "Kevin     David        321 People    Recruitment                                            sourcing,\n",
            "\n",
            "EID-602                                                               456-     kevin.r@educompany.com\n",
            "\n",
            "Roberts   Wilson       Place #405   Specialist                                              interview\n",
            "\n",
            "7894\n",
            "\n",
            "coordination\n",
            "\n",
            "Professional\n",
            "\n",
            "Learning &        (555)\n",
            "\n",
            "Sarah     David       321 People\n",
            "\n",
            "development\n",
            "\n",
            "EID-603                                       Development      456-      sarah.m@educompany.com\n",
            "\n",
            "Mitchell   Wilson       Place#406                                                          programs, training\n",
            "\n",
            "Spec             7895\n",
            "\n",
            "delivery\n",
            "\n",
            "IT Infrastructure Department\n",
            "\n",
            "| Employee<br>ID|Name|Supervisor|Office<br>Location|Position<br>Title|Phone<br>Number|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| Employee<br>ID|Name|Supervisor|Office<br>Location|Position<br>Title|Phone<br>Number|Email Address|Responsibilities|\n",
            "| EID-701|Robert<br>Carter|Robert<br>Taylor|654 Tech<br>Tower #504|IT Director|(555)|robert.c@educompany.com|System<br>architecture,<br>cybersecurity<br>protocols|\n",
            "| EID-701|Robert<br>Carter|Robert<br>Taylor|654 Tech<br>Tower #504|IT Director|567-|robert.c@educompany.com|System<br>architecture,<br>cybersecurity<br>protocols|\n",
            "| EID-701|Robert<br>Carter|Robert<br>Taylor|654 Tech<br>Tower #504|IT Director|8904|robert.c@educompany.com|System<br>architecture,<br>cybersecurity<br>protocols|\n",
            "| EID-701|Jessica<br>Miller|Robert<br>Taylor|654 Tech<br>Tower #505|Network<br>Engineer|(555)|jessica.m@educompany.com|Network<br>infrastructure<br>management|\n",
            "| EID-702|Jessica<br>Miller|Robert<br>Taylor|654 Tech<br>Tower #505|Network<br>Engineer|567-|jessica.m@educompany.com|Network<br>infrastructure<br>management|\n",
            "| EID-702|Jessica<br>Miller|Robert<br>Taylor|654 Tech<br>Tower #505|Network<br>Engineer|8905|jessica.m@educompany.com|Network<br>infrastructure<br>management|\n",
            "| EID-703|Matthew<br>Scott| | | |(555)|matthew.s@educompany.com|Server|\n",
            "| EID-703|Matthew<br>Scott|Robert<br>Taylor|654 Tech<br>Tower#506|Systems|567-|matthew.s@educompany.com|maintenance,|\n",
            "| EID-703|Matthew<br>Scott|Robert<br>Taylor|654 Tech<br>Tower#506|Administrator|8906|matthew.s@educompany.com|system|\n",
            "| EID-703| | | | | |matthew.s@educompany.com|optimization|\n",
            "|  | | | | | |matthew.s@educompany.com| |\n",
            "\n",
            "Performance Management Department\n",
            "\n",
            "| Employee<br>ID| |Supervisor| |Position<br>Title|Phone<br>Number|Email Address|Key|\n",
            "| ---|---|---|---|---|---|---|---|\n",
            "| Employee<br>ID|Name|Supervisor|Office<br>Location|Position<br>Title|Phone<br>Number|Email Address|Responsibilities|\n",
            "| EID-801|Thomas<br>Anderson|David<br>Wilson|321People<br>Place#407|Performance<br>Manager|(555)<br>456-<br>7896|thomas.a@educompany.com|KPI<br>development,<br>evaluation<br>frameworks|\n",
            "\n",
            "Performance\n",
            "\n",
            "(555)\n",
            "\n",
            "Laura       David       321 People    HR Business                                           coaching,\n",
            "\n",
            "EID-802                                                             456-      laura.b@educompany.com\n",
            "\n",
            "Bennett     Wilson       Place#408    Partner\n",
            "\n",
            "compensation\n",
            "\n",
            "7897\n",
            "\n",
            "alignment\n",
            "\n",
            "[empty]\n",
            "(555)                                  Performance\n",
            "\n",
            "Andrew     David       321 People\n",
            "\n",
            "EID-803                                          Data Analyst   456-      andrew.m@educompany.com  metrics analysis,\n",
            "\n",
            "Martin      Wilson       Place#409\n",
            "\n",
            "7898                                  reporting\n",
            "\n",
            "Cross-Departmental Collaboration Framework\n",
            "\n",
            "Our organizational structure emphasizes horizontal collaboration through:\n",
            "\n",
            "1. Project-Based Teams: Temporary task forces combining members from different departments\n",
            "\n",
            "2. Monthly Sync Meetings: Interdepartmental strategy sessions\n",
            "\n",
            "3. Shared KPls: Metrics that require cross-functional achievement\n",
            "\n",
            "Example: The \"Digital Transformation Initiative\" involved:\n",
            "\n",
            "·Instructional Design (curriculum)\n",
            "\n",
            "· Course Development (platform)\n",
            "\n",
            "·IT Infrastructure (system integration)\n",
            "\n",
            "·Marketing (launch strategy)\n",
            "\n",
            "Employee Key Contact Information.md\n",
            "\n",
            "2025-07-10\n",
            "\n",
            "6/ 6\n",
            "\n",
            "· HR (training rollout)\n",
            "\n",
            "Document Version History\n",
            "\n",
            "| Version|Date|Description|Author|\n",
            "| ---|---|---|---|\n",
            "| v1.0|2023-03-15|Initial release|Sarah Johnson|\n",
            "| v1.1|2023-04-01|Added IT department details|Robert Taylor|\n",
            "| v2.0|2023-06-10|Complete restructuring|David Wilson|\n",
            "| v2.1|2023-08-20|Updated contact information|Emily Davis|\n",
            "| v2.2|2023-10-05|Current version|Thomas Anderson|\n",
            "\n",
            "v2.32024-01-15Added duplicate names and expanded Course Development John Smith\n",
            "\n",
            "This document is subject to change without prior notice. For the most current information, please consult the official company intranet portal.\n",
            "\n",
            "\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [
        "# The file is parsed into markdown text that is easy for programs and large models to process via the DashScopeParse interface.\n",
        "def file_to_md(file, category_id):\n",
        "    parse = SilentDashScopeParse(\n",
        "        result_type=ResultType.DASHSCOPE_DOCMIND,\n",
        "        category_id=category_id\n",
        "    )\n",
        "    documents = parse.load_data(file_path=file)\n",
        "    # Initialize an empty string to store Markdown content\n",
        "    markdown_content = \"\"\n",
        "    for doc in documents:\n",
        "        doc_json = json.loads(json.loads(doc.text))\n",
        "        for item in doc_json[\"layouts\"]:\n",
        "            if item[\"text\"] in item[\"markdownContent\"]:\n",
        "                markdown_content += item[\"markdownContent\"]\n",
        "            else:\n",
        "                # When DashScopeParse processes, it will also parse the text information inside document images into the initial markdown text (similar to OCR). This is sufficient for command-line screenshots and text screenshots in the example files of this article. No deep parsing of images is required in this example.\n",
        "                # For actual knowledge base documents, if they involve irregular, complex information in images and require a deeper understanding of the image content, you can call a vision model to further understand the meaning of the image.\n",
        "                # (In the data structure returned by DashScopeParse, for image data, the markdownContent field is the image URL, and the text field is the parsed text.)\n",
        "                # if \".jpg\" in item[\"markdownContent\"] or \".jpeg\" in item[\"markdownContent\"] or \".png\" in item[\"markdownContent\"]:\n",
        "                #     image_url = re.findall(r'\\!\\[.*?\\]\\((https?://.*?)\\)', item[\"markdownContent\"])[0]\n",
        "                #     print(image_url)\n",
        "                #     markdown_content = markdown_content + parse_image_to_text(image_url)+\"\\n\"\n",
        "                # else:\n",
        "                #     markdown_content = markdown_content + item[\"text\"]+\"\\n\"\n",
        "                markdown_content = markdown_content + item[\"text\"]+\"\\n\"\n",
        "    return markdown_content\n",
        "\n",
        "### Example usage\n",
        "\n",
        "# 1. Optional configuration.\n",
        "# On the Bailian platform, different business spaces can be configured for different projects. By default, the default business space is used.\n",
        "# If you need to use a non-default space, go to [Bailian Console - Business Space Management](https://bailian.console.aliyun.com/?admin=1#/efm/business_management), configure the business space, and obtain the Workspace ID.\n",
        "# After completion, uncomment and modify this code to the actual value:\n",
        "# os.environ['DASHSCOPE_WORKSPACE_ID'] = \"<Your Workspace id, Default workspace is empty.>\"\n",
        "\n",
        "# 2. Optional configuration.\n",
        "# When files are parsed through DashScopeParse, the uploaded data directory ID needs to be configured. Go to [Bailian Console - Data Management](https://bailian.console.aliyun.com/#/data-center), configure categories, and obtain the ID.\n",
        "category_id=\"default\" # It is recommended to modify this to a custom category ID for better file classification management.\n",
        "\n",
        "md_content = file_to_md(['./docs/Employee Key Contact Information.pdf'], category_id)\n",
        "print(\"Parsed Markdown text:\")\n",
        "print(\"-\"*100)\n",
        "print(md_content)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7d6da7bb",
      "metadata": {},
      "source": [
        "Due to the diversity of sources for various file formats such as PDF and docx, there may be some minor formatting issues during the process of parsing files into markdown. For example, table rows spanning pages in a PDF might be parsed into multiple lines.\n",
        "\n",
        "LLMs can be used to refine the generated markdown text, correcting issues such as table of contents levels and missing information.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "id": "553aa8a8",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "from dashscope import Generation\n",
        "\n",
        "def md_polisher(data):\n",
        "    messages = [{\n",
        "        'role': 'user',\n",
        "        'content': \n",
        "            '''The following text is converted from PDF to markdown, and there may be some issues with the format and content. I need you to optimize it:\n",
        "                1. Directory levels: If the directory level order is incorrect, please complete or modify it in markdown format;\n",
        "                2. Content errors: If there are inconsistencies in the context, please correct them;\n",
        "                3. Tables: Pay attention to inconsistencies between rows;\n",
        "                4. The overall output should not differ significantly from the input; do not create content on your own—I need to polish the original text;\n",
        "                5. Output format requirement: Markdown text, all your responses should be placed inside a markdown file.\n",
        "                Special Note: Only output the converted markdown content itself, without any other information.\n",
        "                The content to be processed is: \n",
        "            ''' + data\n",
        "        }]\n",
        "    response = Generation.call(\n",
        "        model=\"qwen-plus-0919\",\n",
        "        messages=messages,\n",
        "        result_format='message',\n",
        "        stream=True,\n",
        "        incremental_output=True\n",
        "    )\n",
        "    result = \"\"\n",
        "    print(\"Polished Markdown Text:\")\n",
        "    print(\"-\"*100)\n",
        "    for chunk in response:\n",
        "        print(chunk.output.choices[0].message.content, end='')\n",
        "        result += chunk.output.choices[0].message.content\n",
        "\n",
        "    return(result)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "id": "98a199d7",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Polished Markdown Text:\n",
            "----------------------------------------------------------------------------------------------------\n",
            "```markdown\n",
            "# Employee Key Contact Information\n",
            "\n",
            "## Date: 2025-07-10\n",
            "\n",
            "---\n",
            "\n",
            "## Company Organizational Structure & Key Contact Information\n",
            "\n",
            "### Corporate Mission Statement\n",
            "\n",
            "Our organization is dedicated to pioneering digital transformation in education through innovative learning solutions. As an edtech leader, we specialize in developing interactive online platforms that integrate cutting-edge educational research with emerging technologies. Our strategic vision focuses on three core pillars:\n",
            "\n",
            "1. **Personalized Learning**: Creating adaptive platforms that cater to diverse learning styles and paces\n",
            "2. **Educational Equity**: Ensuring accessible, high-quality resources for all demographics\n",
            "3. **Lifelong Learning**: Developing continuous education pathways that support career advancement\n",
            "\n",
            "We maintain rigorous quality standards through our comprehensive evaluation framework, which includes:\n",
            "\n",
            "- Learner engagement metrics\n",
            "- Pedagogical effectiveness assessments\n",
            "- Technological innovation benchmarks\n",
            "\n",
            "### Departmental Structure Overview\n",
            "\n",
            "#### Interdepartmental\n",
            "\n",
            "| Department Name | Core Responsibilities | Partnerships |\n",
            "| --- | --- | --- |\n",
            "| Instructional Design | Curriculum development, pedagogical research, learning outcomes analysis | Course Development, Evaluation |\n",
            "| Course Development | Technical implementation of educational content, platform integration | IT, Marketing |\n",
            "| Textbook Production | Content curation, exercise design, multimedia resource development | Instructional Design |\n",
            "| Quality Assurance | Content validation, compliance monitoring, learner feedback analysis | All departments |\n",
            "| Marketing & Outreach | Brand positioning, market research, partnership development | Instructional Design |\n",
            "| Human Resources | Talent acquisition, professional development, employee engagement | All departments |\n",
            "| IT Infrastructure | System maintenance, cybersecurity protocols, technical support | All departments |\n",
            "| Performance Management | KPI tracking, 360-degree evaluations, compensation alignment | HR, Finance |\n",
            "\n",
            "### Executive Leadership Team\n",
            "\n",
            "| Name | Position | Office Location | Contact Info |\n",
            "| --- | --- | --- | --- |\n",
            "| Sarah Johnson | Chief Learning Officer (CLO) | 123 Innovation Dr #101 | (555) 123-4567 |\n",
            "| Michael Chen | CTO & Head of Technology | 456 Tech Hub #202 | (555) 234-5678 |\n",
            "| Emily Davis | Director of Marketing | 789 Market Ave #303 | (555) 345-6789 |\n",
            "| David Wilson | VP of Human Resources | 321 People Place #404 | (555) 456-7890 |\n",
            "| Robert Taylor | Director of IT Infrastructure | 654 Tech Tower #505 | (555) 567-8901 |\n",
            "\n",
            "### Departmental Staff Directory\n",
            "\n",
            "#### Instructional Design Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-101 | John Smith | Sarah Johnson | 123 Innovation Dr #101 | Lead Instructional Designer | (555) 123-4567 | john.s@educompany.com | Curriculum architecture, pedagogical strategy, learning analytics |\n",
            "| EID-102 | Maria Gonzalez | Sarah Johnson | 123 Innovation Dr #102 | Senior Instructional Designer | (555) 123-4568 | maria.g@educompany.com | Course optimization, learner engagement analysis, content development |\n",
            "| EID-103 | James Wilson | Sarah Johnson | 123 Innovation Dr #103 | Instructional Designer | (555) 123-4569 | james.w@educompany.com | Multimedia development, content creation |\n",
            "\n",
            "#### Course Development Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-201 | Karen Lee | Michael Chen | 456 Tech Hub #201 | Lead Course Developer | (555) 234-5678 | karen.l@educompany.com | Platform architecture, API integration |\n",
            "| EID-202 | Thomas Nguyen | Michael Chen | 456 Tech Hub #202 | Senior Course Developer | (555) 234-5679 | thomas.n@educompany.com | LMS development, UX optimization |\n",
            "| EID-203 | Olivia Martin | Michael Chen | 456 Tech Hub #203 | Course Developer | (555) 234-5680 | olivia.m@educompany.com | Content migration, system testing |\n",
            "| EID-204 | John Smith | Michael Chen | 456 Tech Hub #204 | Junior Course Developer | (555) 234-5681 | john.smith1@educompany.com | Entry-level tasks, content development |\n",
            "| EID-205 | Michael Johnson | Michael Chen | 456 Tech Hub #205 | QA Specialist | (555) 234-5682 | michael.j@educompany.com | System validation, testing |\n",
            "| EID-206 | Sarah Miller | Michael Chen | 456 Tech Hub #206 | Content Resource Integrator | (555) 234-5683 | sarah.m@educompany.com | Resource coordination, integration |\n",
            "| EID-207 | David Kim | Michael Chen | 456 Tech Hub #207 | UX Designer | (555) 234-5684 | david.k@educompany.com | Interface optimization, user experience |\n",
            "| EID-208 | Emily Davis | Michael Chen | 456 Tech Hub #208 | API Developer | (555) 234-5685 | emily.d@educompany.com | Backend system integration |\n",
            "| EID-209 | Michael Johnson | Michael Chen | 456 Tech Hub #209 | Technical Writer | (555) 234-5686 | michael.johnson@educompany.com | Documentation creation |\n",
            "| EID-210 | Jennifer Lee | Michael Chen | 456 Tech Hub #210 | Data Analyst | (555) 234-5687 | jennifer.l@educompany.com | Performance metrics tracking |\n",
            "| EID-211 | Christopher Lee | Michael Chen | 456 Tech Hub #211 | Security Analyst | (555) 234-5688 | christopher.l@educompany.com | Vulnerability assessments |\n",
            "| EID-212 | Olivia Taylor | Michael Chen | 456 Tech Hub #212 | Project Coordinator | (555) 234-5689 | olivia.t@educompany.com | Timeline management, project coordination |\n",
            "| EID-213 | Daniel Smith | Michael Chen | 456 Tech Hub #213 | Frontend Developer | (555) 234-5690 | daniel.s@educompany.com | UI implementation |\n",
            "| EID-214 | Rachel Kim | Michael Chen | 456 Tech Hub #214 | Mobile App Developer | (555) 234-5691 | rachel.k@educompany.com | Cross-platform development |\n",
            "| EID-215 | Thomas Nguyen | Michael Chen | 456 Tech Hub #215 | Cloud Engineer | (555) 234-5692 | thomas.nguyen@educompany.com | Infrastructure management |\n",
            "| EID-216 | Sophia Li | Michael Chen | 456 Tech Hub #216 | DevOps Specialist | (555) 234-5693 | sophia.l@educompany.com | Continuous integration pipelines |\n",
            "| EID-217 | William Brown | Michael Chen | 456 Tech Hub #217 | Database Administrator | (555) 234-5694 | william.b@educompany.com | Data architecture, database design |\n",
            "| EID-218 | Emma Wilson | Michael Chen | 456 Tech Hub #218 | Technical Support | (555) 234-5695 | emma.w@educompany.com | User assistance |\n",
            "| EID-219 | John Smith | Michael Chen | 456 Tech Hub #219 | Junior QA Tester | (555) 234-5696 | john.smith2@educompany.com | Basic testing procedures |\n",
            "| EID-220 | Michael Johnson | Michael Chen | 456 Tech Hub #220 | System Administrator | (555) 234-5697 | michael.j2@educompany.com | Server maintenance |\n",
            "\n",
            "#### Textbook Production Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-301 | Daniel Kim | Sarah Johnson | 789 Market Ave #301 | Textbook Editor | (555) 345-6789 | daniel.k@educompany.com | Content curation, editorial standards |\n",
            "| EID-302 | Sophia Patel | Sarah Johnson | 789 Market Ave #302 | Senior Textbook Editor | (555) 345-6790 | sophia.p@educompany.com | Multimedia asset coordination, exercise development |\n",
            "| EID-303 | William Harris | Sarah Johnson | 789 Market Ave #303 | Textbook Editor | (555) 345-6791 | william.h@educompany.com | Quality assurance, compliance checks |\n",
            "\n",
            "#### Quality Assurance Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-401 | Emily White | David Wilson | 321 People Place #401 | QA Manager | (555) 456-7890 | emily.w@educompany.com | Testing protocols, certification audits |\n",
            "| EID-402 | Christopher Lee | David Wilson | 321 People Place #402 | QA Analyst | (555) 456-7891 | christopher.l@educompany.com | Learner feedback analysis, compliance monitoring |\n",
            "| EID-403 | Rachel Adams | David Wilson | 321 People Place #403 | QA Specialist | (555) 456-7892 | rachel.a@educompany.com | Content validation, system testing |\n",
            "\n",
            "#### Marketing & Outreach Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-501 | Lisa Chen | Emily Davis | 654 Tech Tower #501 | Marketing Director | (555) 567-8901 | lisa.c@educompany.com | Brand strategy, partnership development |\n",
            "| EID-502 | Mark Thompson | Emily Davis | 654 Tech Tower #502 | Marketing Specialist | (555) 567-8902 | mark.t@educompany.com | Social media campaigns, SEO optimization |\n",
            "| EID-503 | Amanda Walker | Emily Davis | 654 Tech Tower #503 | Content Specialist | (555) 567-8903 | amanda.w@educompany.com | Content curation, thought leadership |\n",
            "\n",
            "#### Human Resources Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-601 | Jennifer Lopez | David Wilson | 321 People Place #404 | HR Director | (555) 456-7893 | jennifer.l@educompany.com | Talent acquisition, employee relations |\n",
            "| EID-602 | Kevin Roberts | David Wilson | 321 People Place #405 | Recruitment Specialist | (555) 456-7894 | kevin.r@educompany.com | Sourcing, interview coordination |\n",
            "| EID-603 | Sarah Mitchell | David Wilson | 321 People Place #406 | Professional Development Specialist | (555) 456-7895 | sarah.m@educompany.com | Training programs, delivery |\n",
            "\n",
            "#### IT Infrastructure Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office Location | Position Title | Phone Number | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-701 | Robert Carter | Robert Taylor | 654 Tech Tower #504 | IT Director | (555) 567-8904 | robert.c@educompany.com | System architecture, cybersecurity protocols |\n",
            "| EID-702 | Jessica Miller | Robert Taylor | 654 Tech Tower #505 | Network Engineer | (555) 567-8905 | jessica.m@educompany.com | Network infrastructure management |\n",
            "| EID-703 | Matthew Scott | Robert Taylor | 654 Tech Tower #506 | Systems Administrator | (555) 567-8906 | matthew.s@educompany.com | Server maintenance, system optimization |\n",
            "\n",
            "#### Performance Management Department\n",
            "\n",
            "| Employee ID | Name | Supervisor | Office Location | Position Title | Phone Number | Email Address | Key Responsibilities |\n",
            "| --- | --- | --- | --- | --- | --- | --- | --- |\n",
            "| EID-801 | Thomas Anderson | David Wilson | 321 People Place #407 | Performance Manager | (555) 456-7896 | thomas.a@educompany.com | KPI development, evaluation frameworks |\n",
            "| EID-802 | Laura Bennett | David Wilson | 321 People Place #408 | HR Business Partner | (555) 456-7897 | laura.b@educompany.com | Coaching, compensation alignment |\n",
            "| EID-803 | Andrew Martin | David Wilson | 321 People Place #409 | Data Analyst | (555) 456-7898 | andrew.m@educompany.com | Performance metrics analysis, reporting |\n",
            "\n",
            "### Cross-Departmental Collaboration Framework\n",
            "\n",
            "Our organizational structure emphasizes horizontal collaboration through:\n",
            "\n",
            "1. **Project-Based Teams**: Temporary task forces combining members from different departments\n",
            "2. **Monthly Sync Meetings**: Interdepartmental strategy sessions\n",
            "3. **Shared KPIs**: Metrics that require cross-functional achievement\n",
            "\n",
            "**Example**: The \"Digital Transformation Initiative\" involved:\n",
            "\n",
            "- Instructional Design (curriculum)\n",
            "- Course Development (platform)\n",
            "- IT Infrastructure (system integration)\n",
            "- Marketing (launch strategy)\n",
            "- HR (training rollout)\n",
            "\n",
            "### Document Version History\n",
            "\n",
            "| Version | Date | Description | Author |\n",
            "| --- | --- | --- | --- |\n",
            "| v1.0 | 2023-03-15 | Initial release | Sarah Johnson |\n",
            "| v1.1 | 2023-04-01 | Added IT department details | Robert Taylor |\n",
            "| v2.0 | 2023-06-10 | Complete restructuring | David Wilson |\n",
            "| v2.1 | 2023-08-20 | Updated contact information | Emily Davis |\n",
            "| v2.2 | 2023-10-05 | Current version | Thomas Anderson |\n",
            "| v2.3 | 2024-01-15 | Added duplicate names and expanded Course Development | John Smith |\n",
            "\n",
            "This document is subject to change without prior notice. For the most current information, please consult the official company intranet portal.\n",
            "```"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'```markdown\\n# Employee Key Contact Information\\n\\n## Date: 2025-07-10\\n\\n---\\n\\n## Company Organizational Structure & Key Contact Information\\n\\n### Corporate Mission Statement\\n\\nOur organization is dedicated to pioneering digital transformation in education through innovative learning solutions. As an edtech leader, we specialize in developing interactive online platforms that integrate cutting-edge educational research with emerging technologies. Our strategic vision focuses on three core pillars:\\n\\n1. **Personalized Learning**: Creating adaptive platforms that cater to diverse learning styles and paces\\n2. **Educational Equity**: Ensuring accessible, high-quality resources for all demographics\\n3. **Lifelong Learning**: Developing continuous education pathways that support career advancement\\n\\nWe maintain rigorous quality standards through our comprehensive evaluation framework, which includes:\\n\\n- Learner engagement metrics\\n- Pedagogical effectiveness assessments\\n- Technological innovation benchmarks\\n\\n### Departmental Structure Overview\\n\\n#### Interdepartmental\\n\\n| Department Name | Core Responsibilities | Partnerships |\\n| --- | --- | --- |\\n| Instructional Design | Curriculum development, pedagogical research, learning outcomes analysis | Course Development, Evaluation |\\n| Course Development | Technical implementation of educational content, platform integration | IT, Marketing |\\n| Textbook Production | Content curation, exercise design, multimedia resource development | Instructional Design |\\n| Quality Assurance | Content validation, compliance monitoring, learner feedback analysis | All departments |\\n| Marketing & Outreach | Brand positioning, market research, partnership development | Instructional Design |\\n| Human Resources | Talent acquisition, professional development, employee engagement | All departments |\\n| IT Infrastructure | System maintenance, cybersecurity protocols, technical support | All departments |\\n| Performance Management | KPI tracking, 360-degree evaluations, compensation alignment | HR, Finance |\\n\\n### Executive Leadership Team\\n\\n| Name | Position | Office Location | Contact Info |\\n| --- | --- | --- | --- |\\n| Sarah Johnson | Chief Learning Officer (CLO) | 123 Innovation Dr #101 | (555) 123-4567 |\\n| Michael Chen | CTO & Head of Technology | 456 Tech Hub #202 | (555) 234-5678 |\\n| Emily Davis | Director of Marketing | 789 Market Ave #303 | (555) 345-6789 |\\n| David Wilson | VP of Human Resources | 321 People Place #404 | (555) 456-7890 |\\n| Robert Taylor | Director of IT Infrastructure | 654 Tech Tower #505 | (555) 567-8901 |\\n\\n### Departmental Staff Directory\\n\\n#### Instructional Design Department\\n\\n| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-101 | John Smith | Sarah Johnson | 123 Innovation Dr #101 | Lead Instructional Designer | (555) 123-4567 | john.s@educompany.com | Curriculum architecture, pedagogical strategy, learning analytics |\\n| EID-102 | Maria Gonzalez | Sarah Johnson | 123 Innovation Dr #102 | Senior Instructional Designer | (555) 123-4568 | maria.g@educompany.com | Course optimization, learner engagement analysis, content development |\\n| EID-103 | James Wilson | Sarah Johnson | 123 Innovation Dr #103 | Instructional Designer | (555) 123-4569 | james.w@educompany.com | Multimedia development, content creation |\\n\\n#### Course Development Department\\n\\n| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-201 | Karen Lee | Michael Chen | 456 Tech Hub #201 | Lead Course Developer | (555) 234-5678 | karen.l@educompany.com | Platform architecture, API integration |\\n| EID-202 | Thomas Nguyen | Michael Chen | 456 Tech Hub #202 | Senior Course Developer | (555) 234-5679 | thomas.n@educompany.com | LMS development, UX optimization |\\n| EID-203 | Olivia Martin | Michael Chen | 456 Tech Hub #203 | Course Developer | (555) 234-5680 | olivia.m@educompany.com | Content migration, system testing |\\n| EID-204 | John Smith | Michael Chen | 456 Tech Hub #204 | Junior Course Developer | (555) 234-5681 | john.smith1@educompany.com | Entry-level tasks, content development |\\n| EID-205 | Michael Johnson | Michael Chen | 456 Tech Hub #205 | QA Specialist | (555) 234-5682 | michael.j@educompany.com | System validation, testing |\\n| EID-206 | Sarah Miller | Michael Chen | 456 Tech Hub #206 | Content Resource Integrator | (555) 234-5683 | sarah.m@educompany.com | Resource coordination, integration |\\n| EID-207 | David Kim | Michael Chen | 456 Tech Hub #207 | UX Designer | (555) 234-5684 | david.k@educompany.com | Interface optimization, user experience |\\n| EID-208 | Emily Davis | Michael Chen | 456 Tech Hub #208 | API Developer | (555) 234-5685 | emily.d@educompany.com | Backend system integration |\\n| EID-209 | Michael Johnson | Michael Chen | 456 Tech Hub #209 | Technical Writer | (555) 234-5686 | michael.johnson@educompany.com | Documentation creation |\\n| EID-210 | Jennifer Lee | Michael Chen | 456 Tech Hub #210 | Data Analyst | (555) 234-5687 | jennifer.l@educompany.com | Performance metrics tracking |\\n| EID-211 | Christopher Lee | Michael Chen | 456 Tech Hub #211 | Security Analyst | (555) 234-5688 | christopher.l@educompany.com | Vulnerability assessments |\\n| EID-212 | Olivia Taylor | Michael Chen | 456 Tech Hub #212 | Project Coordinator | (555) 234-5689 | olivia.t@educompany.com | Timeline management, project coordination |\\n| EID-213 | Daniel Smith | Michael Chen | 456 Tech Hub #213 | Frontend Developer | (555) 234-5690 | daniel.s@educompany.com | UI implementation |\\n| EID-214 | Rachel Kim | Michael Chen | 456 Tech Hub #214 | Mobile App Developer | (555) 234-5691 | rachel.k@educompany.com | Cross-platform development |\\n| EID-215 | Thomas Nguyen | Michael Chen | 456 Tech Hub #215 | Cloud Engineer | (555) 234-5692 | thomas.nguyen@educompany.com | Infrastructure management |\\n| EID-216 | Sophia Li | Michael Chen | 456 Tech Hub #216 | DevOps Specialist | (555) 234-5693 | sophia.l@educompany.com | Continuous integration pipelines |\\n| EID-217 | William Brown | Michael Chen | 456 Tech Hub #217 | Database Administrator | (555) 234-5694 | william.b@educompany.com | Data architecture, database design |\\n| EID-218 | Emma Wilson | Michael Chen | 456 Tech Hub #218 | Technical Support | (555) 234-5695 | emma.w@educompany.com | User assistance |\\n| EID-219 | John Smith | Michael Chen | 456 Tech Hub #219 | Junior QA Tester | (555) 234-5696 | john.smith2@educompany.com | Basic testing procedures |\\n| EID-220 | Michael Johnson | Michael Chen | 456 Tech Hub #220 | System Administrator | (555) 234-5697 | michael.j2@educompany.com | Server maintenance |\\n\\n#### Textbook Production Department\\n\\n| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-301 | Daniel Kim | Sarah Johnson | 789 Market Ave #301 | Textbook Editor | (555) 345-6789 | daniel.k@educompany.com | Content curation, editorial standards |\\n| EID-302 | Sophia Patel | Sarah Johnson | 789 Market Ave #302 | Senior Textbook Editor | (555) 345-6790 | sophia.p@educompany.com | Multimedia asset coordination, exercise development |\\n| EID-303 | William Harris | Sarah Johnson | 789 Market Ave #303 | Textbook Editor | (555) 345-6791 | william.h@educompany.com | Quality assurance, compliance checks |\\n\\n#### Quality Assurance Department\\n\\n| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-401 | Emily White | David Wilson | 321 People Place #401 | QA Manager | (555) 456-7890 | emily.w@educompany.com | Testing protocols, certification audits |\\n| EID-402 | Christopher Lee | David Wilson | 321 People Place #402 | QA Analyst | (555) 456-7891 | christopher.l@educompany.com | Learner feedback analysis, compliance monitoring |\\n| EID-403 | Rachel Adams | David Wilson | 321 People Place #403 | QA Specialist | (555) 456-7892 | rachel.a@educompany.com | Content validation, system testing |\\n\\n#### Marketing & Outreach Department\\n\\n| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-501 | Lisa Chen | Emily Davis | 654 Tech Tower #501 | Marketing Director | (555) 567-8901 | lisa.c@educompany.com | Brand strategy, partnership development |\\n| EID-502 | Mark Thompson | Emily Davis | 654 Tech Tower #502 | Marketing Specialist | (555) 567-8902 | mark.t@educompany.com | Social media campaigns, SEO optimization |\\n| EID-503 | Amanda Walker | Emily Davis | 654 Tech Tower #503 | Content Specialist | (555) 567-8903 | amanda.w@educompany.com | Content curation, thought leadership |\\n\\n#### Human Resources Department\\n\\n| Employee ID | Name | Supervisor | Office | Position Title | Phone | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-601 | Jennifer Lopez | David Wilson | 321 People Place #404 | HR Director | (555) 456-7893 | jennifer.l@educompany.com | Talent acquisition, employee relations |\\n| EID-602 | Kevin Roberts | David Wilson | 321 People Place #405 | Recruitment Specialist | (555) 456-7894 | kevin.r@educompany.com | Sourcing, interview coordination |\\n| EID-603 | Sarah Mitchell | David Wilson | 321 People Place #406 | Professional Development Specialist | (555) 456-7895 | sarah.m@educompany.com | Training programs, delivery |\\n\\n#### IT Infrastructure Department\\n\\n| Employee ID | Name | Supervisor | Office Location | Position Title | Phone Number | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-701 | Robert Carter | Robert Taylor | 654 Tech Tower #504 | IT Director | (555) 567-8904 | robert.c@educompany.com | System architecture, cybersecurity protocols |\\n| EID-702 | Jessica Miller | Robert Taylor | 654 Tech Tower #505 | Network Engineer | (555) 567-8905 | jessica.m@educompany.com | Network infrastructure management |\\n| EID-703 | Matthew Scott | Robert Taylor | 654 Tech Tower #506 | Systems Administrator | (555) 567-8906 | matthew.s@educompany.com | Server maintenance, system optimization |\\n\\n#### Performance Management Department\\n\\n| Employee ID | Name | Supervisor | Office Location | Position Title | Phone Number | Email Address | Key Responsibilities |\\n| --- | --- | --- | --- | --- | --- | --- | --- |\\n| EID-801 | Thomas Anderson | David Wilson | 321 People Place #407 | Performance Manager | (555) 456-7896 | thomas.a@educompany.com | KPI development, evaluation frameworks |\\n| EID-802 | Laura Bennett | David Wilson | 321 People Place #408 | HR Business Partner | (555) 456-7897 | laura.b@educompany.com | Coaching, compensation alignment |\\n| EID-803 | Andrew Martin | David Wilson | 321 People Place #409 | Data Analyst | (555) 456-7898 | andrew.m@educompany.com | Performance metrics analysis, reporting |\\n\\n### Cross-Departmental Collaboration Framework\\n\\nOur organizational structure emphasizes horizontal collaboration through:\\n\\n1. **Project-Based Teams**: Temporary task forces combining members from different departments\\n2. **Monthly Sync Meetings**: Interdepartmental strategy sessions\\n3. **Shared KPIs**: Metrics that require cross-functional achievement\\n\\n**Example**: The \"Digital Transformation Initiative\" involved:\\n\\n- Instructional Design (curriculum)\\n- Course Development (platform)\\n- IT Infrastructure (system integration)\\n- Marketing (launch strategy)\\n- HR (training rollout)\\n\\n### Document Version History\\n\\n| Version | Date | Description | Author |\\n| --- | --- | --- | --- |\\n| v1.0 | 2023-03-15 | Initial release | Sarah Johnson |\\n| v1.1 | 2023-04-01 | Added IT department details | Robert Taylor |\\n| v2.0 | 2023-06-10 | Complete restructuring | David Wilson |\\n| v2.1 | 2023-08-20 | Updated contact information | Emily Davis |\\n| v2.2 | 2023-10-05 | Current version | Thomas Anderson |\\n| v2.3 | 2024-01-15 | Added duplicate names and expanded Course Development | John Smith |\\n\\nThis document is subject to change without prior notice. For the most current information, please consult the official company intranet portal.\\n```'"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "md_polisher(md_content)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ab3e6628",
      "metadata": {},
      "source": [
        "Through the above steps, you have successfully converted the PDF into markdown and made some formatting corrections. If there are images in the document, the information in the images can also be extracted to build a knowledge base that is more conducive to search performance.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "750af73a",
      "metadata": {},
      "source": [
        "#### 4.2.3 Using multiple document chunking methods\n",
        "\n",
        "During the document chunking process, the chunking method can affect the effectiveness of retrieval recall. Let's understand the characteristics of different chunking methods through specific examples. First, create a general evaluation function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "id": "9a58c12b",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "def evaluate_splitter(splitter, documents, question, ground_truth, splitter_name):\n",
        "    \"\"\"Evaluate the effectiveness of different document splitting methods\"\"\"\n",
        "    print(f\"\\n{'='*50}\")\n",
        "    print(f\"🔍 Testing with {splitter_name} method...\")\n",
        "    print(f\"{'='*50}\\n\")\n",
        "\n",
        "    # Build index\n",
        "    print(\"📑 Processing documents...\")\n",
        "    nodes = splitter.get_nodes_from_documents(documents)\n",
        "    index = VectorStoreIndex(nodes, embed_model=Settings.embed_model)\n",
        "\n",
        "    # Create query engine\n",
        "    query_engine = index.as_query_engine(\n",
        "        similarity_top_k=5,\n",
        "        streaming=True\n",
        "    )\n",
        "\n",
        "    # Execute query\n",
        "    print(f\"\\n❓ Test question: {question}\")\n",
        "    print(\"\\n🤖 Model response:\")\n",
        "    response = query_engine.query(question)\n",
        "    response.print_response_stream()\n",
        "\n",
        "    # Output reference snippets\n",
        "    print(f\"\\n📚 Reference snippets recalled by {splitter_name}:\")\n",
        "    for i, node in enumerate(response.source_nodes, 1):\n",
        "        print(f\"\\nDocument snippet {i}:\")\n",
        "        print(\"-\" * 40)\n",
        "        print(node)\n",
        "\n",
        "    # Evaluate results\n",
        "    print(f\"\\n📊 Evaluation results for {splitter_name}:\")\n",
        "    print(\"-\" * 40)\n",
        "    display(evaluate_result(question, response, ground_truth))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "07b76ca6",
      "metadata": {},
      "source": [
        "Next, let's look at the characteristics and examples of various slicing methods:\n",
        "\n",
        "#### 4.2.3.1 Token slicing\n",
        "\n",
        "Suitable for scenarios with strict requirements on the number of tokens, such as when using models with smaller context lengths.\n",
        "\n",
        "Example text: \"LlamaIndex is a powerful RAG framework. It provides various document processing methods. Users can choose the appropriate method based on their needs.\"\n",
        "\n",
        "Possible results after applying token slicing (chunk_size=10):\n",
        "\n",
        "* Slice 1: [\"LlamaIndex\", \"is\", \"a\", \"powerful\", \"RAG\", \"framework.\", \"It\", \"provides\", \"various\", \"document\"]\n",
        "* Slice 2: [\"processing\", \"methods.\", \"Users\", \"can\", \"choose\", \"the\", \"appropriate\", \"method\", \"based\", \"on\"]\n",
        "* Slice 3: [\"their\", \"needs.\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "id": "44f4183a",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================================================\n",
            "🔍 Testing with Token method...\n",
            "==================================================\n",
            "\n",
            "📑 Processing documents...\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "Michael Johnson is in the Course Development department.\n",
            "📚 Reference snippets recalled by Token:\n",
            "\n",
            "Document snippet 1:\n",
            "----------------------------------------\n",
            "Node ID: 2de8f5b2-5e41-4cd1-9c5a-efbf82c91fe7\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document snippet 2:\n",
            "----------------------------------------\n",
            "Node ID: 6580f6cd-64f2-4edb-9f05-ecfafebcd15b\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document snippet 3:\n",
            "----------------------------------------\n",
            "Node ID: c9749b7e-4282-4bfe-98c7-a963a896b702\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.484\n",
            "\n",
            "\n",
            "Document snippet 4:\n",
            "----------------------------------------\n",
            "Node ID: 3f6d716b-e589-459b-9e53-7bc1bf85374a\n",
            "Text: | David Kim        | Michael Chen       | 456 Tech Hub #207 | UX\n",
            "Designer                  | (555) 234-5684   | david.k@educompany.com\n",
            "| Interface optimization\n",
            "| | EID-208     | Emily Davis      | Michael Chen       | 456 Tech Hub\n",
            "#208 | API Developer                | (555) 234...\n",
            "Score:  0.480\n",
            "\n",
            "\n",
            "Document snippet 5:\n",
            "----------------------------------------\n",
            "Node ID: eff51727-1019-440e-8d0c-686bdd45ca6a\n",
            "Text: Kim       | Michael Chen       | 456 Tech Hub #214 | Mobile App\n",
            "Developer         | (555) 234-5691   | rachel.k@educompany.com\n",
            "| Cross-platform development\n",
            "| | EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub\n",
            "#215 | Cloud Engineer               | (555) 234-5692  ...\n",
            "Score:  0.475\n",
            "\n",
            "\n",
            "📊 Evaluation results for Token:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a3bf47f3190f49eab9d0763cc495f42f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is in the Course Development d...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Departmental Structure Overview\\n\\n| Departme...</td>\n",
              "      <td>0.329215</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is in the Course Development d...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Departmental Structure Overview\\n\\n| Departme...            0.329215   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "token_splitter = TokenTextSplitter(\n",
        "    chunk_size=1024,\n",
        "    chunk_overlap=20\n",
        ")\n",
        "evaluate_splitter(token_splitter, documents, question, ground_truth, \"Token\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1955ebb7",
      "metadata": {},
      "source": [
        "#### 4.2.3.2 Sentence slicing\n",
        "\n",
        "This is the default slicing strategy, which preserves the integrity of sentences.\n",
        "\n",
        "The same text after sentence slicing:\n",
        "\n",
        "* Slice 1: \"LlamaIndex is a powerful RAG framework.\"\n",
        "* Slice 2: \"It provides various document processing methods.\"\n",
        "* Slice 3: \"Users can choose the appropriate method based on their needs.\"\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "id": "167f07d0",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================================================\n",
            "🔍 Testing with Sentence method...\n",
            "==================================================\n",
            "\n",
            "📑 Processing documents...\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "Michael Johnson is in the Course Development department.\n",
            "📚 Reference snippets recalled by Sentence:\n",
            "\n",
            "Document snippet 1:\n",
            "----------------------------------------\n",
            "Node ID: 0dab6c4b-0f31-4fd6-b1bd-cbc0bceca1f9\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document snippet 2:\n",
            "----------------------------------------\n",
            "Node ID: 07c77fd3-4809-4a3d-b47e-7156a8a23de6\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document snippet 3:\n",
            "----------------------------------------\n",
            "Node ID: be604d75-5975-49a3-bd85-0a8e2f87d231\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.476\n",
            "\n",
            "\n",
            "Document snippet 4:\n",
            "----------------------------------------\n",
            "Node ID: ef62e1c7-1305-4cf3-beae-41437e47a5cf\n",
            "Text: com         | Resource coordination\n",
            "| | EID-207     | David Kim        | Michael Chen       | 456 Tech Hub\n",
            "#207 | UX Designer                  | (555) 234-5684   |\n",
            "david.k@educompany.com         | Interface optimization\n",
            "| | ...\n",
            "Score:  0.473\n",
            "\n",
            "\n",
            "Document snippet 5:\n",
            "----------------------------------------\n",
            "Node ID: ba7f16cd-e802-41e6-a040-ad2205380304\n",
            "Text: Marketing & Outreach Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "📊 Evaluation results for Sentence:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "48483094c7ce41109235c32ea2dfd5c1",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is in the Course Development d...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Departmental Structure Overview\\n\\n| Departme...</td>\n",
              "      <td>0.342851</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is in the Course Development d...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Departmental Structure Overview\\n\\n| Departme...            0.342851   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence_splitter = SentenceSplitter(\n",
        "    chunk_size=512,\n",
        "    chunk_overlap=50\n",
        ")\n",
        "evaluate_splitter(sentence_splitter, documents, question, ground_truth, \"Sentence\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2eb4052e",
      "metadata": {},
      "source": [
        "#### 4.2.3.3 Sentence window slicing\n",
        "\n",
        "Each slice includes surrounding sentences as the context window.\n",
        "\n",
        "Example text after using sentence window slicing (window_size=1):\n",
        "\n",
        "* Slice 1: \"LlamaIndex is a powerful RAG framework.\" Context: \"It provides various document processing methods.\"\n",
        "* Slice 2: \"It provides various document processing methods.\" Context: \"LlamaIndex is a powerful RAG framework. Users can choose the appropriate method based on their needs.\"\n",
        "* Slice 3: \"Users can choose the appropriate method based on their needs.\" Context: \"It provides various document processing methods.\"\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "id": "4b0394bb",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================================================\n",
            "🔍 Testing with Sentence Window method...\n",
            "==================================================\n",
            "\n",
            "📑 Processing documents...\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "Michael Johnson is in the Course Development department.\n",
            "📚 Reference snippets recalled by Sentence Window:\n",
            "\n",
            "Document snippet 1:\n",
            "----------------------------------------\n",
            "Node ID: 6c125718-4c18-4d0a-8d64-0cb2c0c2fa4b\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document snippet 2:\n",
            "----------------------------------------\n",
            "Node ID: d274d59b-b342-4fc4-8d95-16aae14c938b\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.490\n",
            "\n",
            "\n",
            "Document snippet 3:\n",
            "----------------------------------------\n",
            "Node ID: 86946125-1a12-4242-805b-ef13a4e66f29\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document snippet 4:\n",
            "----------------------------------------\n",
            "Node ID: 58639746-a599-4a2e-bd27-6e2bdd7c2b70\n",
            "Text: Marketing & Outreach Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "Document snippet 5:\n",
            "----------------------------------------\n",
            "Node ID: 2dd44486-cfd9-4153-9e3b-99d37ea7cebf\n",
            "Text: Departmental Staff Directory\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "📊 Evaluation results for Sentence Window:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "fd073233bc37462fb2c7937154ea02ef",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is in the Course Development d...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[\\n\\nDepartmental Structure Overview\\n\\n| Depa...</td>\n",
              "      <td>0.342851</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is in the Course Development d...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [\\n\\nDepartmental Structure Overview\\n\\n| Depa...            0.342851   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.2  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "sentence_window_splitter = SentenceWindowNodeParser.from_defaults(\n",
        "    window_size=3,\n",
        "    window_metadata_key=\"window\",\n",
        "    original_text_metadata_key=\"original_text\"\n",
        ")\n",
        "# Note: Sentence window slicing requires a special post-processor\n",
        "query_engine = index.as_query_engine(\n",
        "    similarity_top_k=5,\n",
        "    streaming=True,\n",
        "    node_postprocessors=[MetadataReplacementPostProcessor(target_metadata_key=\"window\")]\n",
        ")\n",
        "evaluate_splitter(sentence_window_splitter, documents, question, ground_truth, \"Sentence Window\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "48e6c776",
      "metadata": {},
      "source": [
        "#### 4.2.3.4 Semantic chunking\n",
        "\n",
        "Adaptively select chunk points based on semantic relevance.\n",
        "\n",
        "Example text: \"LlamaIndex is a powerful RAG framework. It provides various document processing methods. Users can choose the appropriate method according to their needs. Additionally, it supports vector-based retrieval. This retrieval method is highly efficient.\"\n",
        "\n",
        "Possible results of semantic chunking:\n",
        "\n",
        "* Chunk 1: \"LlamaIndex is a powerful RAG framework. It provides various document processing methods. Users can choose the appropriate method according to their needs.\"\n",
        "* Chunk 2: \"Additionally, it supports vector-based retrieval. This retrieval method is highly efficient.\" (Note that this is grouped by semantic relevance.)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "id": "0e1bb7a4",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================================================\n",
            "🔍 Testing with Semantic method...\n",
            "==================================================\n",
            "\n",
            "📑 Processing documents...\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "Michael Johnson is in the Course Development department.\n",
            "📚 Reference snippets recalled by Semantic:\n",
            "\n",
            "Document snippet 1:\n",
            "----------------------------------------\n",
            "Node ID: 5a699610-0f4c-48f1-ae1c-de870c4814f7\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document snippet 2:\n",
            "----------------------------------------\n",
            "Node ID: eabf545e-a916-4f9a-b0ec-e9ac5c1d9c2b\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.490\n",
            "\n",
            "\n",
            "Document snippet 3:\n",
            "----------------------------------------\n",
            "Node ID: ae770e78-2c93-4678-bdc1-660acdc724db\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document snippet 4:\n",
            "----------------------------------------\n",
            "Node ID: f090e6f0-1a40-4500-a628-9d8b8b75f840\n",
            "Text: Marketing & Outreach Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "Document snippet 5:\n",
            "----------------------------------------\n",
            "Node ID: 0a32808a-36d1-4451-9e7d-b7bc77e63905\n",
            "Text: Departmental Staff Directory\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "📊 Evaluation results for Semantic:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "60c56c422637453dba0efbb4c51bbb40",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is in the Course Development d...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[\\n\\nDepartmental Structure Overview\\n\\n| Depa...</td>\n",
              "      <td>0.342851</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is in the Course Development d...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [\\n\\nDepartmental Structure Overview\\n\\n| Depa...            0.342851   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.2  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "semantic_splitter = SemanticSplitterNodeParser(\n",
        "    buffer_size=1,\n",
        "    breakpoint_percentile_threshold=95,\n",
        "    embed_model=Settings.embed_model\n",
        ")\n",
        "evaluate_splitter(semantic_splitter, documents, question, ground_truth, \"Semantic\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "67722c68",
      "metadata": {},
      "source": [
        "#### 4.2.3.5 Markdown Chunking\n",
        "\n",
        "A chunking method specifically optimized for Markdown documents.\n",
        "\n",
        "Example Markdown text:\n",
        "\n",
        "\n",
        "\n",
        "```markdown\n",
        "# RAG Framework\n",
        "LlamaIndex is a powerful RAG framework.\n",
        "\n",
        "## Features\n",
        "- Provides multiple document processing methods\n",
        "- Supports vector retrieval\n",
        "- Easy and convenient to use\n",
        "\n",
        "### Detailed Description\n",
        "Users can choose the appropriate method based on their needs.\n",
        "```\n",
        "\n",
        "Markdown slices will be intelligently divided based on heading levels:\n",
        "\n",
        "* Slice 1: \"# RAG Framework\n",
        "LlamaIndex is a powerful RAG framework.\"\n",
        "* Slice 2: \"## Features\n",
        "- Provides various document processing methods\n",
        "- Supports vector retrieval\n",
        "- Simple and convenient to use\"\n",
        "* Slice 3: \"### Detailed Description\n",
        "Users can choose the appropriate method according to their needs.\"\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "id": "e28ee065",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================================================\n",
            "🔍 Testing with Markdown method...\n",
            "==================================================\n",
            "\n",
            "📑 Processing documents...\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "Michael Johnson is in the Course Development department.\n",
            "📚 Reference snippets recalled by Markdown:\n",
            "\n",
            "Document snippet 1:\n",
            "----------------------------------------\n",
            "Node ID: 6d946bd8-458e-4e03-bbe3-ee90c8ce9aef\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document snippet 2:\n",
            "----------------------------------------\n",
            "Node ID: 7ec70648-8c6b-41eb-aec4-ef6172492d46\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.490\n",
            "\n",
            "\n",
            "Document snippet 3:\n",
            "----------------------------------------\n",
            "Node ID: 615dc580-e08f-4fde-9233-7ac45112ac7e\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document snippet 4:\n",
            "----------------------------------------\n",
            "Node ID: 690cf6cc-e7ae-4cee-97b4-6f922129c53d\n",
            "Text: Marketing & Outreach Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "Document snippet 5:\n",
            "----------------------------------------\n",
            "Node ID: df6cd7c5-8532-40cb-8777-0e5543e73cb9\n",
            "Text: Departmental Staff Directory\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "📊 Evaluation results for Markdown:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1237d376d40b4b8d97e05357a3efcaa6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is in the Course Development d...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Departmental Structure Overview\\n\\n| Departme...</td>\n",
              "      <td>0.342851</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is in the Course Development d...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Departmental Structure Overview\\n\\n| Departme...            0.342851   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.2  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "markdown_splitter = MarkdownNodeParser()\n",
        "evaluate_splitter(markdown_splitter, documents, question, ground_truth, \"Markdown\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0927b881",
      "metadata": {},
      "source": [
        "In practical applications, there's no need to overthink when choosing a chunking method. You can consider it this way:\n",
        "\n",
        "* If you are new to RAG, it is recommended to start with the default sentence chunking method, which provides good results in most scenarios.\n",
        "* When you find that the retrieval results are not ideal, you can try the following:\n",
        "    * Are you handling long documents, and need to maintain context? Try sentence window chunking.\n",
        "    * Is the document logical and highly specialized? Semantic chunking may be helpful.\n",
        "    * Is the model always reporting token limits exceeded? Token chunking can help you control precisely.\n",
        "    * Processing Markdown documents? Don’t forget there’s dedicated Markdown chunking.\n",
        "\n",
        "There is no single best chunking method—only the one that is most suitable for your specific scenario. Experiment with different chunking methods, observe Ragas evaluation results, and find the solution that best fits your needs. The learning process is all about constant trial and adjustment.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1e35625a",
      "metadata": {},
      "source": [
        "### 4.3 Vectorization and storage phase for slices\n",
        "\n",
        "After document slicing, you also need to index the slices for later retrieval. A common approach is to use a word embedding model to vectorize the slices and store them in a vector database.\n",
        "\n",
        "In this phase, selecting an appropriate word embedding model and vector database is crucial for improving retrieval performance.\n",
        "\n",
        "#### 4.3.1 Understanding word embedding and vectorization\n",
        "\n",
        "The word embedding model can convert text into high-dimensional vectors to represent textual semantics. Similar texts are mapped to  vectors that are close to each other, allowing documents with high similarity to be identified based on the vector representation of a query.\n",
        "\n",
        "_A directed line segment in a plane coordinate system is a 2-dimensional vector. For example, the directed line segment from the origin (0, 0) to point A (xa, ya) can be called vector A. The smaller the angle between vector A and vector B, the higher their similarity._\n",
        "\n",
        "<img src=\"https://img.alicdn.com/imgextra/i4/O1CN01wKAL7C1bhDgbxr2Aa_!!6000000003496-0-tps-1556-1382.jpg\" width=\"400\" ></td>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "id": "71c566ca",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cosine similarity between A and B: 0.987241120712647\n",
            "Cosine similarity between B and C: 0.6050832675335579\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "\n",
        "def cosine_similarity(a, b):\n",
        "    \"\"\"Cosine similarity\"\"\"\n",
        "    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n",
        "\n",
        "# Example vectors\n",
        "a = np.array([0.2, 0.8])\n",
        "b = np.array([0.3, 0.7])\n",
        "c = np.array([0.8, 0.2])\n",
        "\n",
        "print(f\"Cosine similarity between A and B: {cosine_similarity(a, b)}\")\n",
        "print(f\"Cosine similarity between B and C: {cosine_similarity(b, c)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "39489df7",
      "metadata": {},
      "source": [
        "#### 4.3.2 Selecting the appropriate embedding model\n",
        "\n",
        "Different embedding models may produce completely different vectors when calculating the same set of text. Generally, newer embedding models perform better. For example, in the previous section, we used the text-embedding-v2 provided by Alibaba Cloud's Bailian platform. If you switch to a newer version, [text-embedding-v3](https://help.aliyun.com/zh/model-studio/user-guide/embedding), you will notice that even without performing the earlier optimizations,  retrieval performance will still improve to some extent.\n",
        "\n",
        "For example, running the following code shows that different versions of the embedding model yield varying similarity scores for the question \"Which department is Michael Johnson from?\" and different document chunks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "id": "8fc66320",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Query: Which department is Michael Johnson in?\n",
            "Text 1: Course Development Department. Michael Johnson, EID-205, works as a QA Specialist at office location 456 Tech Hub #205. His responsibilities include system validation testing.\n",
            "Text 2: Course Development Department. Michael Johnson, EID-209, serves as a Technical Writer at office location 456 Tech Hub #209. He is responsible for documentation creation.\n",
            "\n",
            "==================== text-embedding-v2 ====================\n",
            "Similarity between query and text 1: 0.4474\n",
            "Similarity between query and text 2: 0.4552\n",
            "\n",
            "==================== text-embedding-v3 ====================\n",
            "Similarity between query and text 1: 0.7239\n",
            "Similarity between query and text 2: 0.7205\n"
          ]
        }
      ],
      "source": [
        "def compare_embeddings(query, chunks, embedding_models):\n",
        "    \"\"\"Compare text similarity across different embedding models\n",
        "\n",
        "    Args:\n",
        "        query: Query text\n",
        "        chunks: List of text chunks to compare\n",
        "        embedding_models: Dictionary of embedding models, format {model_name: model_instance}\n",
        "    \"\"\"\n",
        "    # Print input texts\n",
        "    print(f\"Query: {query}\")\n",
        "    for i, chunk in enumerate(chunks, 1):\n",
        "        print(f\"Text {i}: {chunk}\")\n",
        "\n",
        "    # Calculate and display similarity results for each model\n",
        "    for model_name, model in embedding_models.items():\n",
        "        print(f\"\\n{'='*20} {model_name} {'='*20}\")\n",
        "        query_embedding = (model.get_query_embedding(query) if hasattr(model, 'get_query_embedding')\n",
        "                         else model.get_text_embedding(query))\n",
        "\n",
        "        for i, chunk in enumerate(chunks, 1):\n",
        "            chunk_embedding = model.get_text_embedding(chunk)\n",
        "            similarity = cosine_similarity(query_embedding, chunk_embedding)\n",
        "            print(f\"Similarity between query and text {i}: {similarity:.4f}\")\n",
        "\n",
        "# Prepare test data\n",
        "query = \"Which department is Michael Johnson in?\"\n",
        "chunks = [\n",
        "    # Chunk 1: QA Specialist\n",
        "    \"Course Development Department. Michael Johnson, EID-205, works as a QA Specialist at office location 456 Tech Hub #205. His responsibilities include system validation testing.\",\n",
        "    \n",
        "    # Chunk 2: Technical Writer\n",
        "    \"Course Development Department. Michael Johnson, EID-209, serves as a Technical Writer at office location 456 Tech Hub #209. He is responsible for documentation creation.\"\n",
        "]\n",
        "\n",
        "# Define embedding models to be tested\n",
        "embedding_models = {\n",
        "    \"text-embedding-v2\": DashScopeEmbedding(model_name=\"text-embedding-v2\"),\n",
        "    \"text-embedding-v3\": DashScopeEmbedding(model_name=\"text-embedding-v3\")\n",
        "}\n",
        "\n",
        "# Perform comparison\n",
        "compare_embeddings(query, chunks, embedding_models)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b4b3f761",
      "metadata": {},
      "source": [
        "In addition to evaluating the performance of different embedding models through similarity comparisons, you can also assess them from a practical application perspective. Below, you will use the Ragas evaluation tool to compare the actual performance of the text-embedding-v2 and text-embedding-v3 models within a RAG chatbot.\n",
        "\n",
        "By running the following code, you can clearly see that, under the same RAG chatbot strategy, the overall performance of the text-embedding-v3 model is better than that of text-embedding-v2. Let's take a look at the specific evaluation process and results:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "id": "62c4dcef",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "📑 Processing documents...\n",
            "\n",
            "==================================================\n",
            "🔍 Testing text-embedding-v2...\n",
            "==================================================\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "The provided information does not include a Michael Johnson or specify which department he belongs to. If you have more details or if there's a specific department you're interested in, feel free to let me know!\n",
            "📚 Recalled reference fragments:\n",
            "\n",
            "Document fragment 1:\n",
            "----------------------------------------\n",
            "Node ID: 0de5e0ce-31cd-4e12-8cb1-1e19fed6b355\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.298\n",
            "\n",
            "\n",
            "Document fragment 2:\n",
            "----------------------------------------\n",
            "Node ID: bbb696db-5990-4f7f-9245-d5b80ed0e65c\n",
            "Text: Performance Management Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "-...\n",
            "Score:  0.258\n",
            "\n",
            "\n",
            "Document fragment 3:\n",
            "----------------------------------------\n",
            "Node ID: 4b67cdb3-fd28-4480-9c4a-a5852f36d125\n",
            "Text: Cross-Departmental Collaboration Framework  Our organizational\n",
            "structure emphasizes horizontal collaboration through: 1. **Project-\n",
            "Based Teams**: Temporary task forces combining members from different\n",
            "departments 2. **Monthly Sync Meetings**: Interdepartmental strategy\n",
            "sessions 3. **Shared KPIs**: Metrics that require cross-functional\n",
            "achievemen...\n",
            "Score:  0.246\n",
            "\n",
            "\n",
            "Document fragment 4:\n",
            "----------------------------------------\n",
            "Node ID: 145e8daa-3511-4155-b83b-628e5eb5014f\n",
            "Text: Executive Leadership Team  | Name           | Position\n",
            "| Office Location       | Contact Info                | |-------------\n",
            "---|----------------------------------|------------------------|------\n",
            "-----------------------| | Sarah Johnson  | Chief Learning Officer\n",
            "(CLO)    | 123 Innovation Dr #101 | (555) 123-4567 | sarah.j@e...\n",
            "Score:  0.246\n",
            "\n",
            "\n",
            "Document fragment 5:\n",
            "----------------------------------------\n",
            "Node ID: f0b90a42-ae89-470c-ac9c-1b9319a713a0\n",
            "Text: Departmental Staff Directory\n",
            "Score:  0.245\n",
            "\n",
            "\n",
            "📊 Evaluation results for text-embedding-v2:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9534cea4ffe44bda9dc5b7d08bf865c6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>The provided information does not include a Mi...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Departmental Structure Overview\\n\\n| Departme...</td>\n",
              "      <td>0.377132</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  The provided information does not include a Mi...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Departmental Structure Overview\\n\\n| Departme...            0.377132   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.2  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================================================\n",
            "🔍 Testing text-embedding-v3...\n",
            "==================================================\n",
            "\n",
            "❓ Test question: Which department is Michael Johnson in?\n",
            "\n",
            "🤖 Model response:\n",
            "Michael Johnson is in the Course Development department.\n",
            "📚 Recalled reference fragments:\n",
            "\n",
            "Document fragment 1:\n",
            "----------------------------------------\n",
            "Node ID: 0de5e0ce-31cd-4e12-8cb1-1e19fed6b355\n",
            "Text: Departmental Structure Overview  | Department Name        | Core\n",
            "Responsibilities\n",
            "| Interdepartmental Partnerships | |------------------------|---------\n",
            "----------------------------------------------------------------------\n",
            "-------|-------------------------------| | Instructional Desi...\n",
            "Score:  0.523\n",
            "\n",
            "\n",
            "Document fragment 2:\n",
            "----------------------------------------\n",
            "Node ID: 8198ca96-4048-42d0-a20e-ea6738675cba\n",
            "Text: Course Development Department *(Note: Scroll required for full\n",
            "visibility)*  | Employee ID | Name            | Supervisor         |\n",
            "Office Location | Position Title               | Phone Number     |\n",
            "Email Address                  | Key Responsibilities\n",
            "| |-------------|------------...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document fragment 3:\n",
            "----------------------------------------\n",
            "Node ID: a9422497-734b-44b4-ad5b-8e4aab9d3d5a\n",
            "Text: Instructional Design Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.487\n",
            "\n",
            "\n",
            "Document fragment 4:\n",
            "----------------------------------------\n",
            "Node ID: dae69402-5c65-4fc9-afad-d2932eb12378\n",
            "Text: Marketing & Outreach Department  | Employee ID | Name\n",
            "| Supervisor         | Office Location | Position Title\n",
            "| Phone Number     | Email Address                  | Key\n",
            "Responsibilities\n",
            "| |-------------|------------------|--------------------|-------------\n",
            "---...\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "Document fragment 5:\n",
            "----------------------------------------\n",
            "Node ID: f0b90a42-ae89-470c-ac9c-1b9319a713a0\n",
            "Text: Departmental Staff Directory\n",
            "Score:  0.471\n",
            "\n",
            "\n",
            "📊 Evaluation results for text-embedding-v3:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5f7a2f66d9c7427189da61f8d83c5ae0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Which department is Michael Johnson in?</td>\n",
              "      <td>Michael Johnson is in the Course Development d...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Departmental Structure Overview\\n\\n| Departme...</td>\n",
              "      <td>0.492851</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                  question  \\\n",
              "0  Which department is Michael Johnson in?   \n",
              "\n",
              "                                              answer  \\\n",
              "0  Michael Johnson is in the Course Development d...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Departmental Structure Overview\\n\\n| Departme...            0.492851   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.2  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "def compare_embedding_models(documents, question, ground_truth, sentence_splitter):\n",
        "    \"\"\"Compare the performance of different embedding models in RAG\n",
        "\n",
        "    Args:\n",
        "        documents: List of documents\n",
        "        question: Query question\n",
        "        ground_truth: Standard answer\n",
        "        sentence_splitter: Text splitter\n",
        "    \"\"\"\n",
        "    # Document splitting\n",
        "    print(\"📑 Processing documents...\")\n",
        "    nodes = sentence_splitter.get_nodes_from_documents(documents)\n",
        "\n",
        "    # Define the embedding model configurations to be tested\n",
        "    embedding_models = {\n",
        "        \"text-embedding-v2\": DashScopeEmbedding(\n",
        "            model_name=DashScopeTextEmbeddingModels.TEXT_EMBEDDING_V2\n",
        "        ),\n",
        "        \"text-embedding-v3\": DashScopeEmbedding(\n",
        "            model_name=DashScopeTextEmbeddingModels.TEXT_EMBEDDING_V3,\n",
        "            embed_batch_size=6,\n",
        "            embed_input_length=8192\n",
        "        )\n",
        "    }\n",
        "\n",
        "    # Test each model\n",
        "    for model_name, embed_model in embedding_models.items():\n",
        "        print(f\"\\n{'='*50}\")\n",
        "        print(f\"🔍 Testing {model_name}...\")\n",
        "        print(f\"{'='*50}\")\n",
        "\n",
        "        # Build index and query engine\n",
        "        index = VectorStoreIndex(nodes, embed_model=embed_model)\n",
        "        query_engine = index.as_query_engine(streaming=True, similarity_top_k=5)\n",
        "\n",
        "        # Execute query\n",
        "        print(f\"\\n❓ Test question: {question}\")\n",
        "        print(\"\\n🤖 Model response:\")\n",
        "        response = query_engine.query(question)\n",
        "        response.print_response_stream()\n",
        "\n",
        "        # Display recalled document fragments\n",
        "        print(f\"\\n📚 Recalled reference fragments:\")\n",
        "        for i, node in enumerate(response.source_nodes, 1):\n",
        "            print(f\"\\nDocument fragment {i}:\")\n",
        "            print(\"-\" * 40)\n",
        "            print(node)\n",
        "\n",
        "        # Evaluate results\n",
        "        print(f\"\\n📊 Evaluation results for {model_name}:\")\n",
        "        print(\"-\" * 40)\n",
        "        evaluation_score = evaluate_result(question, response, ground_truth)\n",
        "        display(evaluation_score)\n",
        "\n",
        "# Prepare test data\n",
        "documents = SimpleDirectoryReader('./docs/2_5').load_data()\n",
        "sentence_splitter = SentenceSplitter(\n",
        "    chunk_size=1000,\n",
        "    chunk_overlap=200,\n",
        ")\n",
        "\n",
        "# Perform comparison\n",
        "compare_embedding_models(\n",
        "    documents=documents,\n",
        "    question=question,\n",
        "    ground_truth=ground_truth,\n",
        "    sentence_splitter=sentence_splitter\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3c2a90e3",
      "metadata": {},
      "source": [
        "You can see that:\n",
        "* Newer versions of embedding models generally yield better results (text-embedding-v3 performs better than v2).\n",
        "* In practice, simply upgrading the embedding model can significantly improve retrieval quality.\n",
        "* We recommend you first try the latest text-embedding-v3 model, which delivers good performance across most tasks. Meanwhile, you can keep an eye on updates to DashScope embedding models, and choose to upgrade to a higher-performing version if needed."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "135529fc",
      "metadata": {},
      "source": [
        "#### 4.3.3 Choosing the right vector database\n",
        "\n",
        "When building a RAG chatbot, you have multiple vector storage options to choose from, ranging from simple to complex:\n",
        "\n",
        "##### 4.3.3.1 In-memory vector storage\n",
        "\n",
        "The simplest approach is to use the vector database built into LlamaIndex. Simply install the llama-index package, and with no additional configuration, you can quickly develop and test your RAG chatbot:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "id": "ce24dd06",
      "metadata": {},
      "outputs": [],
      "source": [
        "from llama_index.core import VectorStoreIndex\n",
        "# Create in-memory vector index\n",
        "index = VectorStoreIndex.from_documents(documents)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "208c0bd2",
      "metadata": {},
      "source": [
        "The advantage is that it is quick to get started, making it suitable for development and testing; the disadvantages are that data cannot be persisted, and it is limited by memory size.\n",
        "\n",
        "##### 4.3.3.2 Local vector database\n",
        "\n",
        "When the data volume increases, open-source vector database such as Milvus and Qdrant can be used. These databases provide data persistence and efficient retrieval capabilities."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1cf210cc",
      "metadata": {},
      "source": [
        "The advantage is that the functionality is complete and highly controllable; the disadvantage is that it requires self-deployment and maintenance.\n",
        "\n",
        "##### 4.3.3.3 Cloud service vector storage\n",
        "\n",
        "For production environments, it is recommended to use vector storage capabilities provided by cloud services. Alibaba Cloud offers multiple options:\n",
        "\n",
        "* **Vector Retrieval Service (DashVector)**: Pay-as-you-go, automatic scaling, suitable for quickly starting projects. For detailed functionalities, please refer to [Vector Retrieval Service (DashVector)](https://www.aliyun.com/product/ai/dashvector).\n",
        "* **Vector Retrieval Service Milvus Edition**: Compatible with open-source Milvus, making it convenient to migrate existing applications. For detailed functionalities, please refer to [Vector Retrieval Service Milvus Edition](https://www.aliyun.com/product/milvus).\n",
        "* **Vector Capabilities of Existing Databases**: If you are already using Alibaba Cloud databases (such as RDS and PolarDB), you can  utilize their vector functionalities.\n",
        "\n",
        "The advantages of cloud services include:\n",
        "\n",
        "* With automatic scaling, there's nNo need to worry about operations and maintenance.\n",
        "* Comprehensive monitoring and management tools are provided.\n",
        "* Pay-as-you-go, with better cost control.\n",
        "* Support for hybrid retrieval of vectors + scalars, improving retrieval accuracy.\n",
        "\n",
        "Recommendations:\n",
        "1. Use in-memory vector storage during development and testing.\n",
        "2. For small-scale applications, you can use local vector databases.\n",
        "3. For production environments, it is recommended to use cloud services, and choose the appropriate service type based on your needs.\n",
        "\n",
        "```python\n",
        "import dashvector\n",
        "\n",
        "# Create Client and get collection\n",
        "dashvector_client = dashvector.Client(api_key='YOUR_API_KEY', endpoint='YOUR_CLUSTER_ENDPOINT')\n",
        "collection = dashvector_client.get('quickstart')\n",
        "\n",
        "# Similar vector query\n",
        "collection.query(\n",
        "    vector=[0.1, 0.2, 0.3, 0.4]\n",
        ")\n",
        "\n",
        "# Query using filter conditions\n",
        "collection.query(\n",
        "    vector=[0.1, 0.2, 0.3, 0.4],\n",
        "    topk=100,\n",
        "    filter='age>18',  # Filter condition, only perform similarity search on Docs with age > 18\n",
        "    output_fields=['name', 'age'],  # Only return the name and age fields\n",
        "    include_vector=True\n",
        ")\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4a6b314e",
      "metadata": {},
      "source": [
        "### 4.4 Retrieval recall phase\n",
        "\n",
        "The main issue encountered during the retrieval phase is the difficulty of identifying, from a large number of  slices, the fragment that is most relevant to the user's question and contains the correct answer.\n",
        "\n",
        "From the perspective of intervention timing, solutions can be divided into two main categories:\n",
        "\n",
        "1. Before executing the retrieval, many user queries are incomplete or even ambiguous. It is necessary to find ways to reconstruct the user's true intent to improve retrieval effectiveness.\n",
        "2. After executing the retrieval, you may discover some irrelevant information that needs to be filtered out to avoid interfering with the subsequent answer generation.\n",
        "\n",
        "<table border=\"1\">\n",
        "  <thead>\n",
        "    <tr>\n",
        "      <th>Timing</th>\n",
        "      <th>Improvement Strategy</th>\n",
        "      <th>Example</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <td rowspan=\"7\">Before Retrieval</td>\n",
        "      <td>Question Rewriting</td>\n",
        "      <td>\"Are there any good restaurants nearby?\" => \"Please recommend a few highly-rated restaurants near me.\"</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Question Expansion <em>Adding more information to make the search results more comprehensive</em></td>\n",
        "      <td>\"Which department does Michael Johnson belong to?\" => \"Which department does Michael Johnson belong to? What are his contact details, responsibilities, and work objectives?\"</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Context Expansion Based on User Profile <em>Expanding the question based on user information and behavior data</em></td>\n",
        "      <td>Content Engineer asks \"Work Precautions\" => \"What precautions should a content engineer take at work?\" Project Manager asks \"Work Precautions\" => \"What precautions should a project manager take at work?\"</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Tag Extraction <em>Tag Extraction: Extract tags for subsequent tag filtering + vector similarity search.</em></td>\n",
        "      <td>\"What precautions should a content engineer take at work?\" => <ul><li>Tag Filtering: {\"Position\": \"Content Engineer\"}</li><li>Vector Search: \"What precautions should a content engineer take at work?\"</li></ul></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Ask the User</td>\n",
        "      <td>\"What are the job responsibilities?\" => LLM asks back: \"May I ask which position’s job responsibilities you want to know about?\" <em>Prompt examples for asking back can be found here:</em><a href=\"https://help.aliyun.com/zh/model-studio/use-cases/create-an-ai-shopping-assistant\">Build an AI Shopping Assistant in 10 Minutes</a></td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Think and Plan Multiple Searches</td>\n",
        "      <td>\"Michael Johnson is not available, who can I contact?\" => LLM thinks and plans: => task_1: What are Michael Johnson's responsibilities, task_2: Who else has ${task_1_result} responsibilities => Execute multiple searches in sequence.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td rowspan=\"3\">After Retrieval</td>\n",
        "      <td>reranking + Filtering <em>Most vector databases consider efficiency and sacrifice some accuracy; the retrieved slices may contain items with low relevance.</em></td>\n",
        "      <td>chunk1, chunk2..., chunk10 => chunk 2, chunk4, chunk5</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>Sliding Window Retrieval <em>After retrieving a slice, supplement it with several adjacent slices before and after. This is because adjacent slices often have semantic connections, and looking at a single slice might lose important information.</em> <em>Sliding window retrieval ensures that semantic connections between texts are not lost due to excessive segmentation.</em></td>\n",
        "      <td>A common implementation is sentence sliding windows. You can understand it using the simplified form below: Assume the original text is ABCDEFG (each letter represents a sentence). When the retrieved slice is D, after supplementing adjacent slices, it becomes BCDEF (taking 2 slices before and after). Here, BC and EF are the context of D. For example:<ul><li>BC may contain background information explaining D</li><li>EF may contain subsequent developments or results of D</li><li>These contextual pieces of information help you understand the full meaning of D more accurately</li></ul>By recalling these related context slices, you can improve the accuracy and completeness of the retrieval results.</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <td>...</td>\n",
        "      <td>...</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "18ee79c9",
      "metadata": {},
      "source": [
        "#### 4.4.1 Question rewriting\n",
        "\n",
        "🤔 **Why is question rewriting necessary?**\n",
        "\n",
        "Imagine you are searching for keywords like \"Find Michael Johnson\" or \"Michael Johnson Department.\" It seems simple, but for a RAG system, such scattered search terms can be challenging to process. This is because, in real-world scenarios, there may be multiple employees named Michael Johnson, and the keywords entered by users are often too simplistic, lacking necessary contextual information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "id": "b71bed53",
      "metadata": {},
      "outputs": [],
      "source": [
        "question = \"Find Michael Johnson\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6914f3f9",
      "metadata": {},
      "source": [
        "✨ **What can problem rewriting bring?**\n",
        "\n",
        "Problem rewriting is like helping the system better understand user intent. For example, when you ask \"Find Michael Johnson,\" the system can rewrite the question into a more complete form, such as \"Please tell me all employees named Michael Johnson in the company and their departments.\" Such rewriting improves the accuracy of retrieval, and also makes the answers more comprehensive.\n",
        "\n",
        "Next, you can try different problem rewriting strategies through practical examples. In this case, you will use the following configuration:\n",
        "\n",
        "* Document: Markdown format\n",
        "* Chunking: Default sentence chunking strategy\n",
        "* Model: text-embedding-v3\n",
        "* Storage: Default vector storage\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 54,
      "id": "720fc61c",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Configure embedding model\n",
        "Settings.embed_model = DashScopeEmbedding(\n",
        "    model_name=DashScopeTextEmbeddingModels.TEXT_EMBEDDING_V3,\n",
        "    embed_batch_size=6,\n",
        "    embed_input_length=8192\n",
        ")\n",
        "\n",
        "# Load documents\n",
        "documents = SimpleDirectoryReader('./docs/2_5').load_data()\n",
        "\n",
        "# Configure document splitter\n",
        "sentence_splitter = SentenceSplitter(\n",
        "    chunk_size=1000,\n",
        "    chunk_overlap=200,\n",
        ")\n",
        "\n",
        "# Document splitting\n",
        "sentence_nodes = sentence_splitter.get_nodes_from_documents(documents)\n",
        "\n",
        "# Build index\n",
        "sentence_index = VectorStoreIndex(sentence_nodes, embed_model=Settings.embed_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c0809604",
      "metadata": {},
      "source": [
        "**Conventional Method: Direct Retrieval without Rewriting the Question**\n",
        "\n",
        "Before  attempting to rewrite the question, take a look at the results of using the original question for retrieval. This comparison will give you a more intuitive sense of the improvements that question rewriting can bring:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 55,
      "id": "448c3cc1",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "❓ User question: Find Michael Johnson\n",
            "\n",
            "\n",
            "💭 AI Response:\n",
            "----------------------------------------\n",
            "There is no Michael Johnson listed in the provided contact information for any of the departments. It's possible you may have the wrong name or department. Could you please check the details and try again? If you need help with another name or department, feel free to ask!\n",
            "\n",
            "\n",
            "📚 Reference Sources:\n",
            "----------------------------------------\n",
            "\n",
            "Document snippet 1:\n",
            "Relevance score: 0.4435\n",
            "------------------------------\n",
            "Executive Leadership Team\n",
            "\n",
            "| Name           | Position                        | Office Location       | Contact Info                |\n",
            "|----------------|----------------------------------|------------------------|-----------------------------|\n",
            "| Sarah Johnson  | Chief Learning Officer (CLO)    | 123 Innovation Dr #101 | (555) 123-4567 | sarah.j@educompany.com |\n",
            "| Michael Chen   | CTO & Head of Technology        | 456 Tech Hub #202      | (555) 234-5678 | michael.c@educompany.com |\n",
            "| Emily Davis    | Director of Marketing           | 789 Market Ave #303    | (555) 345-6789 | emily.d@educompany.com |\n",
            "| David Wilson   | VP of Human Resources           | 321 People Place #404  | (555) 456-7890 | david.w@educompany.com |\n",
            "| Robert Taylor  | Director of IT Infrastructure   | 654 Tech Tower #505    | (555) 567-8901 | robert.t@educompany.com |\n",
            "\n",
            "Document snippet 2:\n",
            "Relevance score: 0.4346\n",
            "------------------------------\n",
            "Human Resources Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-601     | Jennifer Lopez   | David Wilson       | 321 People Place #404 | HR Director                  | (555) 456-7893   | jennifer.l@educompany.com      | Talent acquisition, employee relations                                              |\n",
            "| EID-602     | Kevin Roberts    | David Wilson       | 321 People Place #405 | Recruitment Specialist       | (555) 456-7894   | kevin.r@educompany.com         | Candidate sourcing, interview coordination                                          |\n",
            "| EID-603     | Sarah Mitchell   | David Wilson       | 321 People Place #406 | Learning & Development Spec | (555) 456-7895   | sarah.m@educompany.com         | Professional development programs, training delivery                                |\n",
            "\n",
            "Document snippet 3:\n",
            "Relevance score: 0.4323\n",
            "------------------------------\n",
            "Marketing & Outreach Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-501     | Lisa Chen        | Emily Davis        | 654 Tech Tower #501 | Marketing Director           | (555) 567-8901   | lisa.c@educompany.com          | Brand strategy, partnership development                                             |\n",
            "| EID-502     | Mark Thompson    | Emily Davis        | 654 Tech Tower #502 | Digital Marketing Specialist | (555) 567-8902   | mark.t@educompany.com          | Social media campaigns, SEO optimization                                            |\n",
            "| EID-503     | Amanda Walker    | Emily Davis        | 654 Tech Tower #503 | Content Marketing Specialist | (555) 567-8903   | amanda.w@educompany.com        | Educational content curation, thought leadership                                    |\n",
            "\n",
            "Document snippet 4:\n",
            "Relevance score: 0.4310\n",
            "------------------------------\n",
            "Instructional Design Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-101     | John Smith       | Sarah Johnson      | 123 Innovation Dr #101 | Lead Instructional Designer  | (555) 123-4567   | john.s@educompany.com          | Curriculum architecture, pedagogical strategy, learning analytics                     |\n",
            "| EID-102     | Maria Gonzalez   | Sarah Johnson      | 123 Innovation Dr #102 | Senior Instructional Designer| (555) 123-4568   | maria.g@educompany.com         | Course optimization, learner engagement analysis                                      |\n",
            "| EID-103     | James Wilson     | Sarah Johnson      | 123 Innovation Dr #103 | Instructional Designer       | (555) 123-4569   | james.w@educompany.com         | Content development, multimedia integration                                         |\n",
            "\n",
            "Document snippet 5:\n",
            "Relevance score: 0.4276\n",
            "------------------------------\n",
            "Textbook Production Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-301     | Daniel Kim       | Sarah Johnson      | 789 Market Ave #301 | Lead Textbook Editor         | (555) 345-6789   | daniel.k@educompany.com        | Content curation, editorial standards                                               |\n",
            "| EID-302     | Sophia Patel     | Sarah Johnson      | 789 Market Ave #302 | Senior Textbook Editor       | (555) 345-6790   | sophia.p@educompany.com        | Exercise development, multimedia asset coordination                                   |\n",
            "| EID-303     | William Harris   | Sarah Johnson      | 789 Market Ave #303 | Textbook Editor              | (555) 345-6791   | william.h@educompany.com       | Quality assurance, compliance checks                                                |\n",
            "\n",
            "📊 Response Quality Evaluation:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "277eee2d7fad487fbfac2406dbe07cd2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Find Michael Johnson</td>\n",
              "      <td>There is no Michael Johnson listed in the prov...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Executive Leadership Team\\n\\n| Name          ...</td>\n",
              "      <td>0.381531</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               question                                             answer  \\\n",
              "0  Find Michael Johnson  There is no Michael Johnson listed in the prov...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Executive Leadership Team\\n\\n| Name          ...            0.381531   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Create query engine\n",
        "query_engine = sentence_index.as_query_engine(\n",
        "    streaming=True,\n",
        "    similarity_top_k=5\n",
        ")\n",
        "\n",
        "# Execute query\n",
        "print(f\"❓ User question: {question}\\n\")\n",
        "streaming_response = query_engine.query(question)\n",
        "\n",
        "print(\"\\n💭 AI Response:\")\n",
        "print(\"-\" * 40)\n",
        "streaming_response.print_response_stream()\n",
        "print(\"\\n\")\n",
        "\n",
        "# Display reference documents\n",
        "print(\"\\n📚 Reference Sources:\")\n",
        "print(\"-\" * 40)\n",
        "for i, node in enumerate(streaming_response.source_nodes, 1):\n",
        "    print(f\"\\nDocument snippet {i}:\")\n",
        "    print(f\"Relevance score: {node.score:.4f}\")\n",
        "    print(\"-\" * 30)\n",
        "    print(node.text)\n",
        "\n",
        "# Evaluate results\n",
        "print(\"\\n📊 Response Quality Evaluation:\")\n",
        "print(\"-\" * 40)\n",
        "evaluation_score = evaluate_result(question, streaming_response, ground_truth)\n",
        "display(evaluation_score)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8bc446e7",
      "metadata": {},
      "source": [
        "After running this code, you may find the results less than ideal. Although the system retrieved five relevant snippets, it did not find all the information about \"Michael Johnson.\" Why is that?\n",
        "\n",
        "The issue lies in the way the question was asked. When a user asks, \"Find Michael Johnson,\" this question is easy for a person to understand but lacks important context for an LLM—there are multiple Michael Johnsons in the company! This is similar to walking into a company with several employees named John Smith and asking, \"Where is John Smith's desk?\" Someone is likely to respond, \"Which John Smith do you mean?\"\n",
        "\n",
        "So, what if we made the question more complete? For example, by clearly stating that you want to know the department information of \"all employees named Michael Johnson in the company.\" Next, you can try using an LLM to rephrase the question and see if the results improve."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "612e77b9",
      "metadata": {},
      "source": [
        "**Method 1: Using LLMs to Expand User Questions**\n",
        "\n",
        "You can ask the LLM to act as a question-rewriting assistant. It will help you rewrite simple questions to make them more complete and clear. For example, it will not only consider the possibility of multiple individuals named Michael Johnson, but also supplement all related contextual information. Here are the details of how to do it :\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "id": "63552231",
      "metadata": {},
      "outputs": [],
      "source": [
        "query_gen_str = \"\"\"\n",
        "System role setting:\n",
        "You are a professional question rewriting assistant. Your task is to expand the user's original question into a more complete and comprehensive question.\n",
        "\n",
        "Rules:\n",
        "1. Integrate possible ambiguities, related concepts, and contextual information into a complete question\n",
        "2. Use parentheses to supplement explanations for ambiguous concepts\n",
        "3. Add key qualifiers and modifiers\n",
        "4. Ensure that the rewritten question is clear and semantically complete\n",
        "5. For vague concepts, list the main possibilities in parentheses\n",
        "6. About 15 words or less in length\n",
        "\n",
        "Original question:\n",
        "{query}\n",
        "\n",
        "Please generate a comprehensive rewritten question, ensuring:\n",
        "- Contains the core intent of the original question\n",
        "- Covers possible interpretations of ambiguities\n",
        "- Uses clear logical connectives to link different aspects\n",
        "- When necessary, use parentheses to provide supplementary explanations\n",
        "\n",
        "Output format:\n",
        "[Comprehensive rewrite] - The rewritten question\n",
        "\"\"\"\n",
        "query_gen_prompt = PromptTemplate(query_gen_str)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "id": "470dd06f",
      "metadata": {
        "lines_to_next_cell": 1
      },
      "outputs": [],
      "source": [
        "def generate_queries(query: str):\n",
        "    response = Settings.llm.predict(\n",
        "        query_gen_prompt, query=query\n",
        "    )\n",
        "    return response"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 58,
      "id": "1b2a0a37",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "🔍 Original question:\n",
            "   Find Michael Johnson\n",
            "\n",
            "📝 Extended queries:\n",
            "   [Comprehensive rewrite] - Locate Michael Johnson (specific person, athlete, actor) in a specific context or location.\n",
            "\n",
            "💭 AI Response:\n",
            "----------------------------------------\n",
            "There are two individuals named Michael Johnson listed in the provided information:\n",
            "\n",
            "1. **Michael Johnson** - **QA Specialist** in the Course Development Department:\n",
            "   - **Supervisor**: Michael Chen\n",
            "   - **Office Location**: 456 Tech Hub #205\n",
            "   - **Phone Number**: (555) 234-5682\n",
            "   - **Email Address**: michael.j@educompany.com\n",
            "   - **Key Responsibilities**: System validation testing\n",
            "\n",
            "2. **Michael Johnson** - **Technical Writer** in the Course Development Department:\n",
            "   - **Supervisor**: Michael Chen\n",
            "   - **Office Location**: 456 Tech Hub #209\n",
            "   - **Phone Number**: (555) 234-5686\n",
            "   - **Email Address**: michael.johnson@educompany.com\n",
            "   - **Key Responsibilities**: Documentation creation\n",
            "\n",
            "If you are looking for a specific Michael Johnson, such as an athlete or actor, they are not mentioned in this context. The provided information only includes these two individuals within the company's Course Development Department.\n",
            "\n",
            "\n",
            "📚 Reference sources:\n",
            "----------------------------------------\n",
            "\n",
            "Document snippet 1:\n",
            "Relevance score: 0.4444\n",
            "------------------------------\n",
            "Human Resources Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-601     | Jennifer Lopez   | David Wilson       | 321 People Place #404 | HR Director                  | (555) 456-7893   | jennifer.l@educompany.com      | Talent acquisition, employee relations                                              |\n",
            "| EID-602     | Kevin Roberts    | David Wilson       | 321 People Place #405 | Recruitment Specialist       | (555) 456-7894   | kevin.r@educompany.com         | Candidate sourcing, interview coordination                                          |\n",
            "| EID-603     | Sarah Mitchell   | David Wilson       | 321 People Place #406 | Learning & Development Spec | (555) 456-7895   | sarah.m@educompany.com         | Professional development programs, training delivery                                |\n",
            "\n",
            "Document snippet 2:\n",
            "Relevance score: 0.4400\n",
            "------------------------------\n",
            "Performance Management Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-801     | Thomas Anderson  | David Wilson       | 321 People Place #407 | Performance Manager          | (555) 456-7896   | thomas.a@educompany.com        | KPI development, evaluation frameworks                                            |\n",
            "| EID-802     | Laura Bennett    | David Wilson       | 321 People Place #408 | HR Business Partner          | (555) 456-7897   | laura.b@educompany.com         | Performance coaching, compensation alignment                                        |\n",
            "| EID-803     | Andrew Martin    | David Wilson       | 321 People Place #409 | Data Analyst                 | (555) 456-7898   | andrew.m@educompany.com        | Performance metrics analysis, reporting                                             |\n",
            "\n",
            "Document snippet 3:\n",
            "Relevance score: 0.4360\n",
            "------------------------------\n",
            "Executive Leadership Team\n",
            "\n",
            "| Name           | Position                        | Office Location       | Contact Info                |\n",
            "|----------------|----------------------------------|------------------------|-----------------------------|\n",
            "| Sarah Johnson  | Chief Learning Officer (CLO)    | 123 Innovation Dr #101 | (555) 123-4567 | sarah.j@educompany.com |\n",
            "| Michael Chen   | CTO & Head of Technology        | 456 Tech Hub #202      | (555) 234-5678 | michael.c@educompany.com |\n",
            "| Emily Davis    | Director of Marketing           | 789 Market Ave #303    | (555) 345-6789 | emily.d@educompany.com |\n",
            "| David Wilson   | VP of Human Resources           | 321 People Place #404  | (555) 456-7890 | david.w@educompany.com |\n",
            "| Robert Taylor  | Director of IT Infrastructure   | 654 Tech Tower #505    | (555) 567-8901 | robert.t@educompany.com |\n",
            "\n",
            "Document snippet 4:\n",
            "Relevance score: 0.4283\n",
            "------------------------------\n",
            "Departmental Staff Directory\n",
            "\n",
            "Document snippet 5:\n",
            "Relevance score: 0.4162\n",
            "------------------------------\n",
            "Marketing & Outreach Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-501     | Lisa Chen        | Emily Davis        | 654 Tech Tower #501 | Marketing Director           | (555) 567-8901   | lisa.c@educompany.com          | Brand strategy, partnership development                                             |\n",
            "| EID-502     | Mark Thompson    | Emily Davis        | 654 Tech Tower #502 | Digital Marketing Specialist | (555) 567-8902   | mark.t@educompany.com          | Social media campaigns, SEO optimization                                            |\n",
            "| EID-503     | Amanda Walker    | Emily Davis        | 654 Tech Tower #503 | Content Marketing Specialist | (555) 567-8903   | amanda.w@educompany.com        | Educational content curation, thought leadership                                    |\n",
            "\n",
            "Document snippet 6:\n",
            "Relevance score: 0.4130\n",
            "------------------------------\n",
            "Document Version History\n",
            "\n",
            "| Version | Date       | Description                          | Author               |\n",
            "|---------|------------|--------------------------------------|----------------------|\n",
            "| v1.0    | 2023-03-15 | Initial release                      | Sarah Johnson        |\n",
            "| v1.1    | 2023-04-01 | Added IT department details          | Robert Taylor        |\n",
            "| v2.0    | 2023-06-10 | Complete restructuring               | David Wilson         |\n",
            "| v2.1    | 2023-08-20 | Updated contact information          | Emily Davis          |\n",
            "| v2.2    | 2023-10-05 | Current version                      | Thomas Anderson      |\n",
            "| v2.3    | 2024-01-15 | Added duplicate names and expanded Course Development | John Smith       |\n",
            "\n",
            "This document is subject to change without prior notice. For the most current information, please consult the official company intranet portal.\n",
            "\n",
            "Document snippet 7:\n",
            "Relevance score: 0.4089\n",
            "------------------------------\n",
            "Company Organizational Structure & Key Contact Information\n",
            "\n",
            "Document snippet 8:\n",
            "Relevance score: 0.4077\n",
            "------------------------------\n",
            "IT Infrastructure Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-701     | Robert Carter    | Robert Taylor      | 654 Tech Tower #504 | IT Director                  | (555) 567-8904   | robert.c@educompany.com        | System architecture, cybersecurity protocols                                        |\n",
            "| EID-702     | Jessica Miller   | Robert Taylor      | 654 Tech Tower #505 | Network Engineer             | (555) 567-8905   | jessica.m@educompany.com       | Network infrastructure management                                                   |\n",
            "| EID-703     | Matthew Scott    | Robert Taylor      | 654 Tech Tower #506 | Systems Administrator        | (555) 567-8906   | matthew.s@educompany.com       | Server maintenance, system optimization                                             |\n",
            "\n",
            "Document snippet 9:\n",
            "Relevance score: 0.4068\n",
            "------------------------------\n",
            "Instructional Design Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-101     | John Smith       | Sarah Johnson      | 123 Innovation Dr #101 | Lead Instructional Designer  | (555) 123-4567   | john.s@educompany.com          | Curriculum architecture, pedagogical strategy, learning analytics                     |\n",
            "| EID-102     | Maria Gonzalez   | Sarah Johnson      | 123 Innovation Dr #102 | Senior Instructional Designer| (555) 123-4568   | maria.g@educompany.com         | Course optimization, learner engagement analysis                                      |\n",
            "| EID-103     | James Wilson     | Sarah Johnson      | 123 Innovation Dr #103 | Instructional Designer       | (555) 123-4569   | james.w@educompany.com         | Content development, multimedia integration                                         |\n",
            "\n",
            "Document snippet 10:\n",
            "Relevance score: 0.4060\n",
            "------------------------------\n",
            "Course Development Department *(Note: Scroll required for full visibility)*\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-201     | Karen Lee        | Michael Chen       | 456 Tech Hub #201 | Lead Course Developer        | (555) 234-5678   | karen.l@educompany.com         | Platform architecture, API integration                                              |\n",
            "| EID-202     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #202 | Senior Course Developer      | (555) 234-5679   | thomas.n@educompany.com        | LMS development, UX optimization                                                    |\n",
            "| EID-203     | Olivia Martin    | Michael Chen       | 456 Tech Hub #203 | Course Developer             | (555) 234-5680   | olivia.m@educompany.com        | Content migration, system testing                                                   |\n",
            "| EID-204     | John Smith       | Michael Chen       | 456 Tech Hub #204 | Junior Course Developer      | (555) 234-5681   | john.smith1@educompany.com     | Entry-level development tasks                                                       |\n",
            "| EID-205     | Michael Johnson  | Michael Chen       | 456 Tech Hub #205 | QA Specialist                | (555) 234-5682   | michael.j@educompany.com       | System validation testing                                                           |\n",
            "| EID-206     | Sarah Miller     | Michael Chen       | 456 Tech Hub #206 | Content Integrator           | (555) 234-5683   | sarah.m@educompany.com         | Resource coordination                                                               |\n",
            "| EID-207     | David Kim        | Michael Chen       | 456 Tech Hub #207 | UX Designer                  | (555) 234-5684   | david.k@educompany.com         | Interface optimization                                                              |\n",
            "| EID-208     | Emily Davis      | Michael Chen       | 456 Tech Hub #208 | API Developer                | (555) 234-5685   | emily.d@educompany.com         | Backend system integration                                                          |\n",
            "| EID-209     | Michael Johnson  | Michael Chen       | 456 Tech Hub #209 | Technical Writer             | (555) 234-5686   | michael.johnson@educompany.com | Documentation creation                                                              |\n",
            "| EID-210     | Jennifer Lee     | Michael Chen       | 456 Tech Hub #210 | Data Analyst                 | (555) 234-5687   | jennifer.l@educompany.com      | Performance metrics tracking                                                        |\n",
            "| EID-211     | Christopher Lee  | Michael Chen       | 456 Tech Hub #211 | Security Analyst             | (555) 234-5688   | christopher.l@educompany.com   | System vulnerability assessments                                                    |\n",
            "| EID-212     | Olivia Taylor    | Michael Chen       | 456 Tech Hub #212 | Project Coordinator          | (555) 234-5689   | olivia.t@educompany.com        | Timeline management                                                                   |\n",
            "| EID-213     | Daniel Smith     | Michael Chen       | 456 Tech Hub #213 | Frontend Developer           | (555) 234-5690   | daniel.s@educompany.com        | UI implementation                                                                     |\n",
            "| EID-214     | Rachel Kim       | Michael Chen       | 456 Tech Hub #214 | Mobile App Developer         | (555) 234-5691   | rachel.k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.\n",
            "\n",
            "📊 Response quality evaluation:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6e939126aa4e4831b2eaaffb0eda9744",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>[Comprehensive rewrite] - Locate Michael Johns...</td>\n",
              "      <td>There are two individuals named Michael Johnso...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Human Resources Department\\n\\n| Employee ID |...</td>\n",
              "      <td>0.424236</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.267857</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            question  \\\n",
              "0  [Comprehensive rewrite] - Locate Michael Johns...   \n",
              "\n",
              "                                              answer  \\\n",
              "0  There are two individuals named Michael Johnso...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Human Resources Department\\n\\n| Employee ID |...            0.424236   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25           0.267857  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Generate extended queries\n",
        "print(\"\\n🔍 Original question:\")\n",
        "print(f\"   {question}\")\n",
        "query = generate_queries(question)\n",
        "print(\"\\n📝 Extended queries:\")\n",
        "print(f\"   {query}\\n\")\n",
        "\n",
        "# Create query engine\n",
        "query_engine = sentence_index.as_query_engine(\n",
        "    streaming=True,\n",
        "    similarity_top_k=10\n",
        ")\n",
        "# Execute query\n",
        "response = query_engine.query(query)\n",
        "\n",
        "print(\"💭 AI Response:\")\n",
        "print(\"-\" * 40)\n",
        "response.print_response_stream()\n",
        "print(\"\\n\")\n",
        "\n",
        "# Display reference documents\n",
        "print(\"\\n📚 Reference sources:\")\n",
        "print(\"-\" * 40)\n",
        "for i, node in enumerate(response.source_nodes, 1):\n",
        "    print(f\"\\nDocument snippet {i}:\")\n",
        "    print(f\"Relevance score: {node.score:.4f}\")\n",
        "    print(\"-\" * 30)\n",
        "    print(node.text)\n",
        "\n",
        "# Evaluate results\n",
        "print(\"\\n📊 Response quality evaluation:\")\n",
        "print(\"-\" * 40)\n",
        "evaluation_score = evaluate_result(query, response, ground_truth)\n",
        "display(evaluation_score)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ffca5379",
      "metadata": {},
      "source": [
        "After running the code above, you will find that questions rewritten by LLMs can achieve better retrieval results. However, for some complex questions, rewriting alone may not be sufficient."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "bc61fb48",
      "metadata": {},
      "source": [
        "**Method 2: Rewriting a single query into multi-Step queries**\n",
        "\n",
        "In addition to rewriting a query for clarity, you can also break down complex questions into simpler, sequential steps. This approach is particularly useful when dealing with ambiguous or multi-faceted queries—such as identifying information about individuals with common names like \"Michael Johnson.\" By breaking dowjn the original question, you enable the system to retrieve more comprehensive and accurate results.\n",
        "\n",
        "LlamaIndex offers two powerful tools that support this approach:\n",
        "\n",
        "1. `StepDecomposeQueryTransform`: This tool helps break down a complex question into multiple sub-questions. It leverages an LLM to analyze the intent behind the query, and generate a series of logical follow-up questions that lead to the final answer. For the question *\"Which department does Michael Johnson belong to?\"*, the tool might break it down into:\n",
        "    - **Step 1:** *\"How many employees named Michael Johnson are there in the company?\"*\n",
        "    - **Step 2:** *\"Which departments do these Michael Johnsons belong to?\"*\n",
        "\n",
        "By doing so, the system avoids missing relevant information due to ambiguity and ensures all possible matches are considered.\n",
        "\n",
        "2. `MultiStepQueryEngine`: This query engine processes each of the sub-questions generated by `StepDecomposeQueryTransform` in sequence. After retrieving answers to each step, it combines the results into a single, coherent response.\n",
        "    - First, it retrieves all employees named Michael Johnson.\n",
        "    - Then, it queries the department information for each one.\n",
        "    - Finally, it compiles a response such as (just for illustrative purposes):\n",
        "        > *\"There are three Michael Johnsons in the company, working in the Teaching and Research Department, Course Development Department, and IT Department, respectively.\"*\n",
        "\n",
        "This step-by-step processing ensures that no piece of relevant information is overlooked and allows the system to handle complex queries more effectively.\n",
        "\n",
        "**Benefits of multi-step querying**\n",
        "\n",
        "- **Improved Accuracy:** Breaking down queries leads to more thorough retrieval and reduces the risk of missing key details.\n",
        "- **Better Handling of Ambiguity:** Especially useful for queries involving common names or unclear references.\n",
        "- **Logical Flow:** Mimics how humans solve complex problems — by dividing them into smaller, manageable parts.\n",
        "\n",
        "**Considerations**\n",
        "\n",
        "- **Increased LLM Usage:** Since each step may involve calling an LLM, this method consumes more tokens than a single-query approach.\n",
        "- **Longer Processing Time:** Due to multiple sequential calls, the overall response time may be longer.\n",
        "\n",
        "Nonetheless, for complex or ambiguous queries, the trade-off is often worth it in terms of result quality and completeness."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "id": "3168f666",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "❓ User question: Find Michael Johnson\n",
            "\n",
            "🤖 AI is performing multi-step query...\n",
            "\u001b[1;3;33m> Current query: Find Michael Johnson\n",
            "\u001b[0m\u001b[1;3;38;5;200m> New query: Who is Michael Johnson according to the employee key contact information?\n",
            "\u001b[0m\u001b[1;3;33m> Current query: Find Michael Johnson\n",
            "\u001b[0m\u001b[1;3;38;5;200m> New query: Which Michael Johnson is being referred to in the request, the QA Specialist or the Technical Writer?\n",
            "\u001b[0m\u001b[1;3;33m> Current query: Find Michael Johnson\n",
            "\u001b[0m\u001b[1;3;38;5;200m> New query: Which Michael Johnson is being referred to in the request, the QA Specialist or the Technical Writer?\n",
            "\u001b[0m\n",
            "📚 Reference basis:\n",
            "----------------------------------------\n",
            "\n",
            "Document fragment 1:\n",
            "------------------------------\n",
            "\n",
            "Question: Who is Michael Johnson according to the employee key contact information?\n",
            "Answer: There are two individuals named Michael Johnson listed in the employee key contact information:\n",
            "\n",
            "1. One works as a QA Specialist in the Course Development Department, supervised by Michael Chen, located at 456 Tech Hub #205, with the phone number (555) 234-5682 and email michael.j@educompany.com.\n",
            "\n",
            "2. The other is a Technical Writer in the same department, also supervised by Michael Chen, located at 456 Tech Hub #209, with the phone number (555) 234-5686 and email michael.johnson@educompany.com.\n",
            "\n",
            "Document fragment 2:\n",
            "------------------------------\n",
            "\n",
            "Question: Which Michael Johnson is being referred to in the request, the QA Specialist or the Technical Writer?\n",
            "Answer: The Michael Johnson who is a QA Specialist has the email address michael.j@educompany.com, while the Michael Johnson who is a Technical Writer has the email address michael.johnson@educompany.com. To determine which one is being referred to in the request, you would need to check the specific role mentioned in the request. If the request mentions a QA Specialist, then it refers to the one with the email michael.j@educompany.com. If it mentions a Technical Writer, then it refers to the one with the email michael.johnson@educompany.com.\n",
            "\n",
            "Document fragment 3:\n",
            "------------------------------\n",
            "\n",
            "Question: Which Michael Johnson is being referred to in the request, the QA Specialist or the Technical Writer?\n",
            "Answer: The Michael Johnson referred to as the QA Specialist has the email address michael.j@educompany.com, while the Michael Johnson referred to as the Technical Writer has the email address michael.johnson@educompany.com. To determine which one is being referred to in the request, you would need to check the specific role mentioned or the context of the communication.\n",
            "\n",
            "Document fragment 4:\n",
            "------------------------------\n",
            "Human Resources Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-601     | Jennifer Lopez   | David Wilson       | 321 People Place #404 | HR Director                  | (555) 456-7893   | jennifer.l@educompany.com      | Talent acquisition, employee relations                                              |\n",
            "| EID-602     | Kevin Roberts    | David Wilson       | 321 People Place #405 | Recruitment Specialist       | (555) 456-7894   | kevin.r@educompany.com         | Candidate sourcing, interview coordination                                          |\n",
            "| EID-603     | Sarah Mitchell   | David Wilson       | 321 People Place #406 | Learning & Development Spec | (555) 456-7895   | sarah.m@educompany.com         | Professional development programs, training delivery                                |\n",
            "\n",
            "Document fragment 5:\n",
            "------------------------------\n",
            "Executive Leadership Team\n",
            "\n",
            "| Name           | Position                        | Office Location       | Contact Info                |\n",
            "|----------------|----------------------------------|------------------------|-----------------------------|\n",
            "| Sarah Johnson  | Chief Learning Officer (CLO)    | 123 Innovation Dr #101 | (555) 123-4567 | sarah.j@educompany.com |\n",
            "| Michael Chen   | CTO & Head of Technology        | 456 Tech Hub #202      | (555) 234-5678 | michael.c@educompany.com |\n",
            "| Emily Davis    | Director of Marketing           | 789 Market Ave #303    | (555) 345-6789 | emily.d@educompany.com |\n",
            "| David Wilson   | VP of Human Resources           | 321 People Place #404  | (555) 456-7890 | david.w@educompany.com |\n",
            "| Robert Taylor  | Director of IT Infrastructure   | 654 Tech Tower #505    | (555) 567-8901 | robert.t@educompany.com |\n",
            "\n",
            "Document fragment 6:\n",
            "------------------------------\n",
            "Marketing & Outreach Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-501     | Lisa Chen        | Emily Davis        | 654 Tech Tower #501 | Marketing Director           | (555) 567-8901   | lisa.c@educompany.com          | Brand strategy, partnership development                                             |\n",
            "| EID-502     | Mark Thompson    | Emily Davis        | 654 Tech Tower #502 | Digital Marketing Specialist | (555) 567-8902   | mark.t@educompany.com          | Social media campaigns, SEO optimization                                            |\n",
            "| EID-503     | Amanda Walker    | Emily Davis        | 654 Tech Tower #503 | Content Marketing Specialist | (555) 567-8903   | amanda.w@educompany.com        | Educational content curation, thought leadership                                    |\n",
            "\n",
            "Document fragment 7:\n",
            "------------------------------\n",
            "Course Development Department *(Note: Scroll required for full visibility)*\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-201     | Karen Lee        | Michael Chen       | 456 Tech Hub #201 | Lead Course Developer        | (555) 234-5678   | karen.l@educompany.com         | Platform architecture, API integration                                              |\n",
            "| EID-202     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #202 | Senior Course Developer      | (555) 234-5679   | thomas.n@educompany.com        | LMS development, UX optimization                                                    |\n",
            "| EID-203     | Olivia Martin    | Michael Chen       | 456 Tech Hub #203 | Course Developer             | (555) 234-5680   | olivia.m@educompany.com        | Content migration, system testing                                                   |\n",
            "| EID-204     | John Smith       | Michael Chen       | 456 Tech Hub #204 | Junior Course Developer      | (555) 234-5681   | john.smith1@educompany.com     | Entry-level development tasks                                                       |\n",
            "| EID-205     | Michael Johnson  | Michael Chen       | 456 Tech Hub #205 | QA Specialist                | (555) 234-5682   | michael.j@educompany.com       | System validation testing                                                           |\n",
            "| EID-206     | Sarah Miller     | Michael Chen       | 456 Tech Hub #206 | Content Integrator           | (555) 234-5683   | sarah.m@educompany.com         | Resource coordination                                                               |\n",
            "| EID-207     | David Kim        | Michael Chen       | 456 Tech Hub #207 | UX Designer                  | (555) 234-5684   | david.k@educompany.com         | Interface optimization                                                              |\n",
            "| EID-208     | Emily Davis      | Michael Chen       | 456 Tech Hub #208 | API Developer                | (555) 234-5685   | emily.d@educompany.com         | Backend system integration                                                          |\n",
            "| EID-209     | Michael Johnson  | Michael Chen       | 456 Tech Hub #209 | Technical Writer             | (555) 234-5686   | michael.johnson@educompany.com | Documentation creation                                                              |\n",
            "| EID-210     | Jennifer Lee     | Michael Chen       | 456 Tech Hub #210 | Data Analyst                 | (555) 234-5687   | jennifer.l@educompany.com      | Performance metrics tracking                                                        |\n",
            "| EID-211     | Christopher Lee  | Michael Chen       | 456 Tech Hub #211 | Security Analyst             | (555) 234-5688   | christopher.l@educompany.com   | System vulnerability assessments                                                    |\n",
            "| EID-212     | Olivia Taylor    | Michael Chen       | 456 Tech Hub #212 | Project Coordinator          | (555) 234-5689   | olivia.t@educompany.com        | Timeline management                                                                   |\n",
            "| EID-213     | Daniel Smith     | Michael Chen       | 456 Tech Hub #213 | Frontend Developer           | (555) 234-5690   | daniel.s@educompany.com        | UI implementation                                                                     |\n",
            "| EID-214     | Rachel Kim       | Michael Chen       | 456 Tech Hub #214 | Mobile App Developer         | (555) 234-5691   | rachel.k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.\n",
            "\n",
            "Document fragment 8:\n",
            "------------------------------\n",
            "IT Infrastructure Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-701     | Robert Carter    | Robert Taylor      | 654 Tech Tower #504 | IT Director                  | (555) 567-8904   | robert.c@educompany.com        | System architecture, cybersecurity protocols                                        |\n",
            "| EID-702     | Jessica Miller   | Robert Taylor      | 654 Tech Tower #505 | Network Engineer             | (555) 567-8905   | jessica.m@educompany.com       | Network infrastructure management                                                   |\n",
            "| EID-703     | Matthew Scott    | Robert Taylor      | 654 Tech Tower #506 | Systems Administrator        | (555) 567-8906   | matthew.s@educompany.com       | Server maintenance, system optimization                                             |\n",
            "\n",
            "Document fragment 9:\n",
            "------------------------------\n",
            "Quality Assurance Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-401     | Emily White      | David Wilson       | 321 People Place #401 | QA Manager                   | (555) 456-7890   | emily.w@educompany.com         | Testing protocols, certification audits                                             |\n",
            "| EID-402     | Christopher Lee  | David Wilson       | 321 People Place #402 | QA Analyst                   | (555) 456-7891   | christopher.l@educompany.com   | Learner feedback analysis, compliance monitoring                                    |\n",
            "| EID-403     | Rachel Adams     | David Wilson       | 321 People Place #403 | QA Specialist                | (555) 456-7892   | rachel.a@educompany.com        | Content validation, system testing                                                  |\n",
            "\n",
            "Document fragment 10:\n",
            "------------------------------\n",
            "k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.com       | Data architecture design                                                              |\n",
            "| EID-218     | Emma Wilson      | Michael Chen       | 456 Tech Hub #218 | Technical Support            | (555) 234-5695   | emma.w@educompany.com          | User assistance                                                                         |\n",
            "| EID-219     | John Smith       | Michael Chen       | 456 Tech Hub #219 | Junior QA Tester             | (555) 234-5696   | john.smith2@educompany.com     | Basic testing procedures                                                              |\n",
            "| EID-220     | Michael Johnson  | Michael Chen       | 456 Tech Hub #220 | System Administrator         | (555) 234-5697   | michael.j2@educompany.com      | Server maintenance                                                                    |\n",
            "\n",
            "Document fragment 11:\n",
            "------------------------------\n",
            "Textbook Production Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-301     | Daniel Kim       | Sarah Johnson      | 789 Market Ave #301 | Lead Textbook Editor         | (555) 345-6789   | daniel.k@educompany.com        | Content curation, editorial standards                                               |\n",
            "| EID-302     | Sophia Patel     | Sarah Johnson      | 789 Market Ave #302 | Senior Textbook Editor       | (555) 345-6790   | sophia.p@educompany.com        | Exercise development, multimedia asset coordination                                   |\n",
            "| EID-303     | William Harris   | Sarah Johnson      | 789 Market Ave #303 | Textbook Editor              | (555) 345-6791   | william.h@educompany.com       | Quality assurance, compliance checks                                                |\n",
            "\n",
            "Document fragment 12:\n",
            "------------------------------\n",
            "Course Development Department *(Note: Scroll required for full visibility)*\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-201     | Karen Lee        | Michael Chen       | 456 Tech Hub #201 | Lead Course Developer        | (555) 234-5678   | karen.l@educompany.com         | Platform architecture, API integration                                              |\n",
            "| EID-202     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #202 | Senior Course Developer      | (555) 234-5679   | thomas.n@educompany.com        | LMS development, UX optimization                                                    |\n",
            "| EID-203     | Olivia Martin    | Michael Chen       | 456 Tech Hub #203 | Course Developer             | (555) 234-5680   | olivia.m@educompany.com        | Content migration, system testing                                                   |\n",
            "| EID-204     | John Smith       | Michael Chen       | 456 Tech Hub #204 | Junior Course Developer      | (555) 234-5681   | john.smith1@educompany.com     | Entry-level development tasks                                                       |\n",
            "| EID-205     | Michael Johnson  | Michael Chen       | 456 Tech Hub #205 | QA Specialist                | (555) 234-5682   | michael.j@educompany.com       | System validation testing                                                           |\n",
            "| EID-206     | Sarah Miller     | Michael Chen       | 456 Tech Hub #206 | Content Integrator           | (555) 234-5683   | sarah.m@educompany.com         | Resource coordination                                                               |\n",
            "| EID-207     | David Kim        | Michael Chen       | 456 Tech Hub #207 | UX Designer                  | (555) 234-5684   | david.k@educompany.com         | Interface optimization                                                              |\n",
            "| EID-208     | Emily Davis      | Michael Chen       | 456 Tech Hub #208 | API Developer                | (555) 234-5685   | emily.d@educompany.com         | Backend system integration                                                          |\n",
            "| EID-209     | Michael Johnson  | Michael Chen       | 456 Tech Hub #209 | Technical Writer             | (555) 234-5686   | michael.johnson@educompany.com | Documentation creation                                                              |\n",
            "| EID-210     | Jennifer Lee     | Michael Chen       | 456 Tech Hub #210 | Data Analyst                 | (555) 234-5687   | jennifer.l@educompany.com      | Performance metrics tracking                                                        |\n",
            "| EID-211     | Christopher Lee  | Michael Chen       | 456 Tech Hub #211 | Security Analyst             | (555) 234-5688   | christopher.l@educompany.com   | System vulnerability assessments                                                    |\n",
            "| EID-212     | Olivia Taylor    | Michael Chen       | 456 Tech Hub #212 | Project Coordinator          | (555) 234-5689   | olivia.t@educompany.com        | Timeline management                                                                   |\n",
            "| EID-213     | Daniel Smith     | Michael Chen       | 456 Tech Hub #213 | Frontend Developer           | (555) 234-5690   | daniel.s@educompany.com        | UI implementation                                                                     |\n",
            "| EID-214     | Rachel Kim       | Michael Chen       | 456 Tech Hub #214 | Mobile App Developer         | (555) 234-5691   | rachel.k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.\n",
            "\n",
            "Document fragment 13:\n",
            "------------------------------\n",
            "IT Infrastructure Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-701     | Robert Carter    | Robert Taylor      | 654 Tech Tower #504 | IT Director                  | (555) 567-8904   | robert.c@educompany.com        | System architecture, cybersecurity protocols                                        |\n",
            "| EID-702     | Jessica Miller   | Robert Taylor      | 654 Tech Tower #505 | Network Engineer             | (555) 567-8905   | jessica.m@educompany.com       | Network infrastructure management                                                   |\n",
            "| EID-703     | Matthew Scott    | Robert Taylor      | 654 Tech Tower #506 | Systems Administrator        | (555) 567-8906   | matthew.s@educompany.com       | Server maintenance, system optimization                                             |\n",
            "\n",
            "Document fragment 14:\n",
            "------------------------------\n",
            "Quality Assurance Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-401     | Emily White      | David Wilson       | 321 People Place #401 | QA Manager                   | (555) 456-7890   | emily.w@educompany.com         | Testing protocols, certification audits                                             |\n",
            "| EID-402     | Christopher Lee  | David Wilson       | 321 People Place #402 | QA Analyst                   | (555) 456-7891   | christopher.l@educompany.com   | Learner feedback analysis, compliance monitoring                                    |\n",
            "| EID-403     | Rachel Adams     | David Wilson       | 321 People Place #403 | QA Specialist                | (555) 456-7892   | rachel.a@educompany.com        | Content validation, system testing                                                  |\n",
            "\n",
            "Document fragment 15:\n",
            "------------------------------\n",
            "k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.com       | Data architecture design                                                              |\n",
            "| EID-218     | Emma Wilson      | Michael Chen       | 456 Tech Hub #218 | Technical Support            | (555) 234-5695   | emma.w@educompany.com          | User assistance                                                                         |\n",
            "| EID-219     | John Smith       | Michael Chen       | 456 Tech Hub #219 | Junior QA Tester             | (555) 234-5696   | john.smith2@educompany.com     | Basic testing procedures                                                              |\n",
            "| EID-220     | Michael Johnson  | Michael Chen       | 456 Tech Hub #220 | System Administrator         | (555) 234-5697   | michael.j2@educompany.com      | Server maintenance                                                                    |\n",
            "\n",
            "Document fragment 16:\n",
            "------------------------------\n",
            "Textbook Production Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-301     | Daniel Kim       | Sarah Johnson      | 789 Market Ave #301 | Lead Textbook Editor         | (555) 345-6789   | daniel.k@educompany.com        | Content curation, editorial standards                                               |\n",
            "| EID-302     | Sophia Patel     | Sarah Johnson      | 789 Market Ave #302 | Senior Textbook Editor       | (555) 345-6790   | sophia.p@educompany.com        | Exercise development, multimedia asset coordination                                   |\n",
            "| EID-303     | William Harris   | Sarah Johnson      | 789 Market Ave #303 | Textbook Editor              | (555) 345-6791   | william.h@educompany.com       | Quality assurance, compliance checks                                                |\n",
            "\n",
            "Document fragment 17:\n",
            "------------------------------\n",
            "Course Development Department *(Note: Scroll required for full visibility)*\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-201     | Karen Lee        | Michael Chen       | 456 Tech Hub #201 | Lead Course Developer        | (555) 234-5678   | karen.l@educompany.com         | Platform architecture, API integration                                              |\n",
            "| EID-202     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #202 | Senior Course Developer      | (555) 234-5679   | thomas.n@educompany.com        | LMS development, UX optimization                                                    |\n",
            "| EID-203     | Olivia Martin    | Michael Chen       | 456 Tech Hub #203 | Course Developer             | (555) 234-5680   | olivia.m@educompany.com        | Content migration, system testing                                                   |\n",
            "| EID-204     | John Smith       | Michael Chen       | 456 Tech Hub #204 | Junior Course Developer      | (555) 234-5681   | john.smith1@educompany.com     | Entry-level development tasks                                                       |\n",
            "| EID-205     | Michael Johnson  | Michael Chen       | 456 Tech Hub #205 | QA Specialist                | (555) 234-5682   | michael.j@educompany.com       | System validation testing                                                           |\n",
            "| EID-206     | Sarah Miller     | Michael Chen       | 456 Tech Hub #206 | Content Integrator           | (555) 234-5683   | sarah.m@educompany.com         | Resource coordination                                                               |\n",
            "| EID-207     | David Kim        | Michael Chen       | 456 Tech Hub #207 | UX Designer                  | (555) 234-5684   | david.k@educompany.com         | Interface optimization                                                              |\n",
            "| EID-208     | Emily Davis      | Michael Chen       | 456 Tech Hub #208 | API Developer                | (555) 234-5685   | emily.d@educompany.com         | Backend system integration                                                          |\n",
            "| EID-209     | Michael Johnson  | Michael Chen       | 456 Tech Hub #209 | Technical Writer             | (555) 234-5686   | michael.johnson@educompany.com | Documentation creation                                                              |\n",
            "| EID-210     | Jennifer Lee     | Michael Chen       | 456 Tech Hub #210 | Data Analyst                 | (555) 234-5687   | jennifer.l@educompany.com      | Performance metrics tracking                                                        |\n",
            "| EID-211     | Christopher Lee  | Michael Chen       | 456 Tech Hub #211 | Security Analyst             | (555) 234-5688   | christopher.l@educompany.com   | System vulnerability assessments                                                    |\n",
            "| EID-212     | Olivia Taylor    | Michael Chen       | 456 Tech Hub #212 | Project Coordinator          | (555) 234-5689   | olivia.t@educompany.com        | Timeline management                                                                   |\n",
            "| EID-213     | Daniel Smith     | Michael Chen       | 456 Tech Hub #213 | Frontend Developer           | (555) 234-5690   | daniel.s@educompany.com        | UI implementation                                                                     |\n",
            "| EID-214     | Rachel Kim       | Michael Chen       | 456 Tech Hub #214 | Mobile App Developer         | (555) 234-5691   | rachel.k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.\n",
            "\n",
            "Document fragment 18:\n",
            "------------------------------\n",
            "IT Infrastructure Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-701     | Robert Carter    | Robert Taylor      | 654 Tech Tower #504 | IT Director                  | (555) 567-8904   | robert.c@educompany.com        | System architecture, cybersecurity protocols                                        |\n",
            "| EID-702     | Jessica Miller   | Robert Taylor      | 654 Tech Tower #505 | Network Engineer             | (555) 567-8905   | jessica.m@educompany.com       | Network infrastructure management                                                   |\n",
            "| EID-703     | Matthew Scott    | Robert Taylor      | 654 Tech Tower #506 | Systems Administrator        | (555) 567-8906   | matthew.s@educompany.com       | Server maintenance, system optimization                                             |\n",
            "\n",
            "📊 Multi-step query evaluation results:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "15d832d996b64876810ab3298ae491e8",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Find Michael Johnson</td>\n",
              "      <td>There are two individuals named Michael Johnso...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[\\nQuestion: Who is Michael Johnson according ...</td>\n",
              "      <td>0.347734</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               question                                             answer  \\\n",
              "0  Find Michael Johnson  There are two individuals named Michael Johnso...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [\\nQuestion: Who is Michael Johnson according ...            0.347734   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0             0.0                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from llama_index.core.indices.query.query_transform.base import (\n",
        "    StepDecomposeQueryTransform,\n",
        ")\n",
        "step_decompose_transform = StepDecomposeQueryTransform(verbose=True)\n",
        "# set Logging to DEBUG for more detailed outputs\n",
        "from llama_index.core.query_engine import MultiStepQueryEngine\n",
        "query_engine = sentence_index.as_query_engine(streaming=True,similarity_top_k=5)\n",
        "query_engine = MultiStepQueryEngine(\n",
        "    query_engine=query_engine,\n",
        "    query_transform=step_decompose_transform,\n",
        "    index_summary=\"Employee Key Contact Information\"\n",
        ")\n",
        "print(f\"❓ User question: {question}\\n\")\n",
        "print(\"🤖 AI is performing multi-step query...\")\n",
        "response = query_engine.query(question)\n",
        "print(\"\\n📚 Reference basis:\")\n",
        "print(\"-\" * 40)\n",
        "for i, node in enumerate(response.source_nodes, 1):\n",
        "    print(f\"\\nDocument fragment {i}:\")\n",
        "    print(\"-\" * 30)\n",
        "    print(node.text)\n",
        "\n",
        "# Evaluation results\n",
        "print(\"\\n📊 Multi-step query evaluation results:\")\n",
        "print(\"-\" * 40)\n",
        "evaluation_score = evaluate_result(question, response, ground_truth)\n",
        "display(evaluation_score)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "04def672",
      "metadata": {},
      "source": [
        "In this way, the system begins by understanding the overall intent behind the query, then breaks it down into a series of smaller, sequential steps to resolve ambiguity and gather accurate information. For example, when the user asks the LLM to \"Find Michael Johnson,\" it might first attempt to clarify the request by asking:\n",
        "\n",
        "```\n",
        "\"Who is Michael Johnson according to the employee key contact information?\"\n",
        "```\n",
        "\n",
        "Based on the retrieved information, the system may then follow up with a more specific question such as:\n",
        "\n",
        "```\n",
        "\"Which Michael Johnson is being referred to in the request — the QA Specialist or the Technical Writer?\"\n",
        "```\n",
        "\n",
        "This step-by-step approach ensures that all possible matches are considered and helps the system avoid making assumptions. Once the correct individual is identified, the system can then proceed to retrieve detailed information, such as their department, contact details, or supervisor.\n",
        "\n",
        "By processing queries in this structured manner, the system significantly improves the accuracy and relevance of its responses, especially when dealing with ambiguous or incomplete questions."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ad0d5034",
      "metadata": {},
      "source": [
        "**Method 3: Enhance retrieval with HyDE**\n",
        "\n",
        "The previous methods have all been about adjusting the question itself. Now, let's try a different approach: what if we first assume a possible answer? This is the unique aspect of the Hypothetical Document Embeddings  (HyDE) method.\n",
        "\n",
        "Here is its working mechanism:\n",
        "\n",
        "1. First, have the LLM generate a \"hypothetical answer document\" based on the question.\n",
        "2. Use this hypothetical document to retrieve real documents.\n",
        "3. Finally, use the retrieved real documents to generate an actual answer.\n",
        "\n",
        "This is analagous to when you're looking for a book and already have a rough outline of its content in mind, then use that outline to match similar books in the library. Let's see how this can be implemented specifically:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "id": "d2f49b65",
      "metadata": {
        "lines_to_next_cell": 2
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "❓ User question: Find Michael Johnson\n",
            "\n",
            "🤖 AI is analyzing using HyDE...\n",
            "\n",
            "💭 AI response:\n",
            "----------------------------------------\n",
            "There are two individuals named Michael Johnson listed:\n",
            "\n",
            "1. **Michael Johnson**  \n",
            "   - **Position:** QA Specialist  \n",
            "   - **Supervisor:** Michael Chen  \n",
            "   - **Office Location:** 456 Tech Hub #205  \n",
            "   - **Phone Number:** (555) 234-5682  \n",
            "   - **Email Address:** michael.j@educompany.com  \n",
            "   - **Key Responsibilities:** System validation testing  \n",
            "\n",
            "2. **Michael Johnson**  \n",
            "   - **Position:** Technical Writer  \n",
            "   - **Supervisor:** Michael Chen  \n",
            "   - **Office Location:** 456 Tech Hub #209  \n",
            "   - **Phone Number:** (555) 234-5686  \n",
            "   - **Email Address:** michael.johnson@educompany.com  \n",
            "   - **Key Responsibilities:** Documentation creation\n",
            "📚 Reference sources:\n",
            "----------------------------------------\n",
            "\n",
            "Document snippet 1:\n",
            "------------------------------\n",
            "Performance Management Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-801     | Thomas Anderson  | David Wilson       | 321 People Place #407 | Performance Manager          | (555) 456-7896   | thomas.a@educompany.com        | KPI development, evaluation frameworks                                            |\n",
            "| EID-802     | Laura Bennett    | David Wilson       | 321 People Place #408 | HR Business Partner          | (555) 456-7897   | laura.b@educompany.com         | Performance coaching, compensation alignment                                        |\n",
            "| EID-803     | Andrew Martin    | David Wilson       | 321 People Place #409 | Data Analyst                 | (555) 456-7898   | andrew.m@educompany.com        | Performance metrics analysis, reporting                                             |\n",
            "\n",
            "Document snippet 2:\n",
            "------------------------------\n",
            "Executive Leadership Team\n",
            "\n",
            "| Name           | Position                        | Office Location       | Contact Info                |\n",
            "|----------------|----------------------------------|------------------------|-----------------------------|\n",
            "| Sarah Johnson  | Chief Learning Officer (CLO)    | 123 Innovation Dr #101 | (555) 123-4567 | sarah.j@educompany.com |\n",
            "| Michael Chen   | CTO & Head of Technology        | 456 Tech Hub #202      | (555) 234-5678 | michael.c@educompany.com |\n",
            "| Emily Davis    | Director of Marketing           | 789 Market Ave #303    | (555) 345-6789 | emily.d@educompany.com |\n",
            "| David Wilson   | VP of Human Resources           | 321 People Place #404  | (555) 456-7890 | david.w@educompany.com |\n",
            "| Robert Taylor  | Director of IT Infrastructure   | 654 Tech Tower #505    | (555) 567-8901 | robert.t@educompany.com |\n",
            "\n",
            "Document snippet 3:\n",
            "------------------------------\n",
            "Marketing & Outreach Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-501     | Lisa Chen        | Emily Davis        | 654 Tech Tower #501 | Marketing Director           | (555) 567-8901   | lisa.c@educompany.com          | Brand strategy, partnership development                                             |\n",
            "| EID-502     | Mark Thompson    | Emily Davis        | 654 Tech Tower #502 | Digital Marketing Specialist | (555) 567-8902   | mark.t@educompany.com          | Social media campaigns, SEO optimization                                            |\n",
            "| EID-503     | Amanda Walker    | Emily Davis        | 654 Tech Tower #503 | Content Marketing Specialist | (555) 567-8903   | amanda.w@educompany.com        | Educational content curation, thought leadership                                    |\n",
            "\n",
            "Document snippet 4:\n",
            "------------------------------\n",
            "Course Development Department *(Note: Scroll required for full visibility)*\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-201     | Karen Lee        | Michael Chen       | 456 Tech Hub #201 | Lead Course Developer        | (555) 234-5678   | karen.l@educompany.com         | Platform architecture, API integration                                              |\n",
            "| EID-202     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #202 | Senior Course Developer      | (555) 234-5679   | thomas.n@educompany.com        | LMS development, UX optimization                                                    |\n",
            "| EID-203     | Olivia Martin    | Michael Chen       | 456 Tech Hub #203 | Course Developer             | (555) 234-5680   | olivia.m@educompany.com        | Content migration, system testing                                                   |\n",
            "| EID-204     | John Smith       | Michael Chen       | 456 Tech Hub #204 | Junior Course Developer      | (555) 234-5681   | john.smith1@educompany.com     | Entry-level development tasks                                                       |\n",
            "| EID-205     | Michael Johnson  | Michael Chen       | 456 Tech Hub #205 | QA Specialist                | (555) 234-5682   | michael.j@educompany.com       | System validation testing                                                           |\n",
            "| EID-206     | Sarah Miller     | Michael Chen       | 456 Tech Hub #206 | Content Integrator           | (555) 234-5683   | sarah.m@educompany.com         | Resource coordination                                                               |\n",
            "| EID-207     | David Kim        | Michael Chen       | 456 Tech Hub #207 | UX Designer                  | (555) 234-5684   | david.k@educompany.com         | Interface optimization                                                              |\n",
            "| EID-208     | Emily Davis      | Michael Chen       | 456 Tech Hub #208 | API Developer                | (555) 234-5685   | emily.d@educompany.com         | Backend system integration                                                          |\n",
            "| EID-209     | Michael Johnson  | Michael Chen       | 456 Tech Hub #209 | Technical Writer             | (555) 234-5686   | michael.johnson@educompany.com | Documentation creation                                                              |\n",
            "| EID-210     | Jennifer Lee     | Michael Chen       | 456 Tech Hub #210 | Data Analyst                 | (555) 234-5687   | jennifer.l@educompany.com      | Performance metrics tracking                                                        |\n",
            "| EID-211     | Christopher Lee  | Michael Chen       | 456 Tech Hub #211 | Security Analyst             | (555) 234-5688   | christopher.l@educompany.com   | System vulnerability assessments                                                    |\n",
            "| EID-212     | Olivia Taylor    | Michael Chen       | 456 Tech Hub #212 | Project Coordinator          | (555) 234-5689   | olivia.t@educompany.com        | Timeline management                                                                   |\n",
            "| EID-213     | Daniel Smith     | Michael Chen       | 456 Tech Hub #213 | Frontend Developer           | (555) 234-5690   | daniel.s@educompany.com        | UI implementation                                                                     |\n",
            "| EID-214     | Rachel Kim       | Michael Chen       | 456 Tech Hub #214 | Mobile App Developer         | (555) 234-5691   | rachel.k@educompany.com        | Cross-platform development                                                            |\n",
            "| EID-215     | Thomas Nguyen    | Michael Chen       | 456 Tech Hub #215 | Cloud Engineer               | (555) 234-5692   | thomas.nguyen@educompany.com   | Infrastructure management                                                             |\n",
            "| EID-216     | Sophia Li        | Michael Chen       | 456 Tech Hub #216 | DevOps Specialist            | (555) 234-5693   | sophia.l@educompany.com        | Continuous integration pipelines                                                    |\n",
            "| EID-217     | William Brown    | Michael Chen       | 456 Tech Hub #217 | Database Administrator       | (555) 234-5694   | william.b@educompany.\n",
            "\n",
            "Document snippet 5:\n",
            "------------------------------\n",
            "IT Infrastructure Department\n",
            "\n",
            "| Employee ID | Name            | Supervisor         | Office Location | Position Title               | Phone Number     | Email Address                  | Key Responsibilities                                                                 |\n",
            "|-------------|------------------|--------------------|------------------|-------------------------------|------------------|--------------------------------|---------------------------------------------------------------------------------------|\n",
            "| EID-701     | Robert Carter    | Robert Taylor      | 654 Tech Tower #504 | IT Director                  | (555) 567-8904   | robert.c@educompany.com        | System architecture, cybersecurity protocols                                        |\n",
            "| EID-702     | Jessica Miller   | Robert Taylor      | 654 Tech Tower #505 | Network Engineer             | (555) 567-8905   | jessica.m@educompany.com       | Network infrastructure management                                                   |\n",
            "| EID-703     | Matthew Scott    | Robert Taylor      | 654 Tech Tower #506 | Systems Administrator        | (555) 567-8906   | matthew.s@educompany.com       | Server maintenance, system optimization                                             |\n",
            "\n",
            "📊 HyDE Query Evaluation Results:\n",
            "----------------------------------------\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "3b4e266359c44486be0a49deb9c235fb",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Find Michael Johnson</td>\n",
              "      <td>There are two individuals named Michael Johnso...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Performance Management Department\\n\\n| Employ...</td>\n",
              "      <td>0.733581</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               question                                             answer  \\\n",
              "0  Find Michael Johnson  There are two individuals named Michael Johnso...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Performance Management Department\\n\\n| Employ...            0.733581   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from llama_index.core.indices.query.query_transform.base import (\n",
        "    HyDEQueryTransform,\n",
        ")\n",
        "from llama_index.core.query_engine import TransformQueryEngine\n",
        "# run query with HyDE query transform\n",
        "hyde = HyDEQueryTransform(include_original=True)\n",
        "query_engine = sentence_index.as_query_engine(streaming=True,similarity_top_k=5)\n",
        "query_engine = TransformQueryEngine(query_engine, query_transform=hyde)\n",
        "\n",
        "print(f\"❓ User question: {question}\\n\")\n",
        "print(\"🤖 AI is analyzing using HyDE...\")\n",
        "streaming_response = query_engine.query(question)\n",
        "\n",
        "print(\"\\n💭 AI response:\")\n",
        "print(\"-\" * 40)\n",
        "streaming_response.print_response_stream()\n",
        "\n",
        "# Display reference documents\n",
        "print(\"\\n📚 Reference sources:\")\n",
        "print(\"-\" * 40)\n",
        "for i, node in enumerate(streaming_response.source_nodes, 1):\n",
        "    print(f\"\\nDocument snippet {i}:\")\n",
        "    print(\"-\" * 30)\n",
        "    print(node.text)\n",
        "\n",
        "# Evaluate results\n",
        "print(\"\\n📊 HyDE Query Evaluation Results:\")\n",
        "print(\"-\" * 40)\n",
        "evaluation_score = evaluate_result(question, streaming_response, ground_truth)\n",
        "display(evaluation_score)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11d22721",
      "metadata": {},
      "source": [
        "As you can see from the evaluation results, this method has indeed brought some improvements. You may be wondering: How does the system generate this \"hypothetical document\"? Let’s take a look at what content the LLM  generated during this process:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 65,
      "id": "f76a564d",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "🤖 AI-generated hypothetical document:\n",
            "Certainly! Here is the passage with the requested information:\n",
            "\n",
            "---\n",
            "\n",
            "To find Michael Johnson in the employee key contact information table, you should look for the following details:\n",
            "\n",
            "**Name:** Michael Johnson  \n",
            "**Position:** Senior Software Engineer  \n",
            "**Department:** Technology  \n",
            "**Office Location:** Building 3, Floor 5, Room 502  \n",
            "**Phone Number:** (555) 123-4567  \n",
            "**Email Address:** michael.johnson@company.com  \n",
            "**Employee ID:** 12345  \n",
            "**Supervisor:** Jane Doe  \n",
            "**Team:** Development Team 2  \n",
            "**Start Date:** June 1, 2018  \n",
            "**Emergency Contact:** Sarah Johnson, (555) 987-6543\n",
            "\n",
            "Michael Johnson is a key member of the Technology department, specifically within the Development Team 2. He has been with the company since June 1, 2018, and can be reached at his office in Building 3, Floor 5, Room 502. For any urgent matters, his emergency contact is Sarah Johnson.\n",
            "\n",
            "---\n",
            "\n",
            "This passage includes all the key details that would help locate and contact Michael Johnson effectively.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "query_bundle = hyde(question + \"from the employee key contact information table\")\n",
        "hyde_doc = query_bundle.embedding_strs[0]\n",
        "print(f\"🤖 AI-generated hypothetical document:\\n{hyde_doc}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1fe30925",
      "metadata": {},
      "source": [
        "Although this \"hypothetical document\" is entirely fabricated by AI, its structure and style are very similar to real company employee information. LlamaIndex provides flexible control mechanisms to optimize this process:\n",
        "The HyDEQueryTransform class lets us precisely control the generation of hypothetical documents in the following ways:\n",
        "\n",
        "* Custom LLM: By passing different configurations of LLMs through the llm parameter, you can choose a more suitable language model for generating hypothetical documents.\n",
        "* Prompt template: Customize the prompt template via the hyde_prompt parameter to precisely control the format and content of the output.\n",
        "* Query strategy: Use the include_original parameter to decide whether to combine the original query with the hypothetical document.\n",
        "\n",
        "TransformQueryEngine acts as a wrapper for the query engine, which will:\n",
        "\n",
        "1. First call HyDEQueryTransform to generate the hypothetical document.\n",
        "2. Use the hypothetical document for vector retrieval.\n",
        "3. Return the query results.\n",
        "\n",
        "This architecture lets us optimize the retrieval effect by adjusting the parameters of HyDEQueryTransform without modifying the underlying query engine. Even though the specific content of the hypothetical document may not be entirely accurate, a well-designed configuration can help the system retrieve relevant information more accurately."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9d9325c8",
      "metadata": {},
      "source": [
        "#### 4.4.2 Extracting tags to enhance retrieval\n",
        "\n",
        "On the basis of vector retrieval, we can also add tag filtering to improve retrieval accuracy. This method is similar to a library having both title search and a classification numbering system, which allows for more precise retrieval.\n",
        "\n",
        "There are two key scenarios for tag extraction:\n",
        "\n",
        "1. When building an index, extract structured tags from document slices.\n",
        "2. During retrieval, extract corresponding tags from user queries for filtering.\n",
        "\n",
        "Let's look at two examples to understand how to extract tags from different types of text:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 66,
      "id": "c1fe9eca",
      "metadata": {},
      "outputs": [],
      "source": [
        "import os\n",
        "from openai import OpenAI\n",
        "client = OpenAI(api_key=os.getenv(\"DASHSCOPE_API_KEY\"), base_url=\"https://dashscope.aliyuncs.com/compatible-mode/v1\")\n",
        "system_message = \"\"\"You are a tag extraction expert. Please extract structured information from the text and output tags as required.\n",
        "---\n",
        "[Supported Tag Types]\n",
        "- Person Name\n",
        "- Department Name\n",
        "- Job Title\n",
        "- Technical Field\n",
        "- Product Name\n",
        "---\n",
        "[Output Requirements]\n",
        "1. Please output in JSON format, such as: [{\"key\": \"Department Name\", \"value\": \"Teaching and Research Department\"}]\n",
        "2. If a certain type of tag is not identified, do not output that type\n",
        "---\n",
        "The text to be analyzed is as follows:\n",
        "\"\"\"\n",
        "def extract_tags(text):\n",
        "    completion = client.chat.completions.create(\n",
        "        model=\"qwen-turbo\",\n",
        "        messages=[\n",
        "            {'role': 'system', 'content': system_message},\n",
        "            {'role': 'user', 'content': text}\n",
        "        ],\n",
        "        response_format={\"type\": \"json_object\"}\n",
        "    )\n",
        "    return completion.choices[0].message.content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 67,
      "id": "90889dbc",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "HR Document Tag Extraction Results:\n",
            "[{\"key\": \"Person Name\", \"value\": \"David Miller\"}, {\"key\": \"Department Name\", \"value\": \"AI Research and Development department\"}, {\"key\": \"Job Title\", \"value\": \"Technical Director\"}, {\"key\": \"Technical Field\", \"value\": \"natural language processing\"}, {\"key\": \"Product Name\", \"value\": \"AstraChat\"}]\n",
            "\n",
            "Technical Document Tag Extraction Results:\n",
            "[{\"key\": \"Technical Field\", \"value\": \"deep learning-based image recognition algorithm\"}, {\"key\": \"Department Name\", \"value\": \"CT diagnostic system\"}, {\"key\": \"Product Name\", \"value\": \"Johns Hopkins Hospital\"}]\n"
          ]
        }
      ],
      "source": [
        "# Example 1: HR Document\n",
        "hr_text = \"\"\"David Miller is the Technical Director of our AI Research and Development department. He led the team in building the next-generation intelligent conversational platform, AstraChat, and has extensive experience in the field of natural language processing. For more information about the project, feel free to contact him directly.\"\"\"\n",
        "print(\"HR Document Tag Extraction Results:\")\n",
        "print(extract_tags(hr_text))\n",
        "\n",
        "# Example 2: Technical Document\n",
        "tech_text = \"\"\"This paper introduces a deep learning-based image recognition algorithm that has achieved significant breakthroughs in medical imaging analysis. The algorithm has been deployed in the CT diagnostic system at Johns Hopkins Hospital.\"\"\"\n",
        "print(\"\\nTechnical Document Tag Extraction Results:\")\n",
        "print(extract_tags(tech_text))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "241b1c05",
      "metadata": {},
      "source": [
        "When we build the index, we can store these tags alongside document chunks. This way, during retrieval, for example, when a user asks \"Which department is Michael Johnson in?\" we can:\n",
        "\n",
        "1. Extract the name tag {\"key\": \"Name\", \"value\": \"Michael Johnson\"} from the question.\n",
        "2. Use the tag to filter out all document chunks containing \"Michael Johnson.\"\n",
        "3. Apply vector similarity search to find the most relevant content.\n",
        "\n",
        "This combination of \"tag filtering + vector retrieval\" significantly improves retrieval accuracy. It performs especially well when dealing with highly structured enterprise documents.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ce4befa3",
      "metadata": {},
      "source": [
        "#### 4.4.3 Re-ranking\n",
        "\n",
        "You can delete the Markdown file created earlier to recreate the initial poor response state for the query \"Find Michael Johnson\" mentioned at the beginning of this section."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "id": "8d7125c3",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Folder has been deleted.\n"
          ]
        }
      ],
      "source": [
        "![ -d ./docs/2_5/ ] && rm -r ./docs/2_5/ && echo \"Folder has been deleted.\" || echo \"Folder does not exist, no need to delete.\""
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d6af44f1",
      "metadata": {},
      "source": [
        "Then, execute the following code. As you can see, the code is set to retrieve three relevant document slices from the vector database.\n",
        "\n",
        "However, the retrieval results are not sufficient—one entry for \"Michael Johnson\" is missing. Due to this incomplete recall, the Q&A bot is unable to correctly respond to the request \"Find Michael Johnson,\" resulting in an incorrect response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 84,
      "id": "2b39d4b7",
      "metadata": {},
      "outputs": [],
      "source": [
        "from llama_index.llms.dashscope import DashScope\n",
        "from chatbot import rag"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 85,
      "id": "baa75114",
      "metadata": {},
      "outputs": [],
      "source": [
        "index = rag.create_index('./docs')\n",
        "query_engine = index.as_query_engine(\n",
        "    similarity_top_k=3,\n",
        "    streaming=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 86,
      "id": "0c424b6f",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "==================================================\n",
            "🤔 Question: Find Michael Johnson\n",
            "==================================================\n",
            "\n",
            "🤖 Answer:\n",
            "There are two employees named Michael Johnson in the provided information:\n",
            "\n",
            "1. **Michael Johnson**  \n",
            "   - **Employee ID:** EID-220  \n",
            "   - **Supervisor:** Michael Chen  \n",
            "   - **Office Location:** 456 Tech Hub #220  \n",
            "   - **Position Title:** System Administrator  \n",
            "   - **Phone Number:** (555) 234-5697  \n",
            "   - **Email Address:** michael.j2@educompany.com  \n",
            "   - **Key Responsibilities:** Server maintenance  \n",
            "\n",
            "2. **Michael Johnson**  \n",
            "   - **Employee ID:** EID-205  \n",
            "   - **Supervisor:** Michael Chen  \n",
            "   - **Office Location:** 456 Tech Hub #205  \n",
            "   - **Position Title:** QA Specialist  \n",
            "   - **Phone Number:** (555) 234-5682  \n",
            "   - **Email Address:** michael.j@educompany.com  \n",
            "   - **Key Responsibilities:** System validation testing  \n",
            "\n",
            "3. **Michael Johnson**  \n",
            "   - **Employee ID:** EID-209  \n",
            "   - **Supervisor:** Michael Chen  \n",
            "   - **Office Location:** 456 Tech Hub #209  \n",
            "   - **Position Title:** Technical Writer  \n",
            "   - **Phone Number:** (555) 234-5686  \n",
            "   - **Email Address:** michael.johnson@educompany.com  \n",
            "   - **Key Responsibilities:** Documentation creation  \n",
            "\n",
            "If you need more specific information about any of these Michael Johnsons, please let me know!\n",
            "--------------------------------------------------\n",
            "📚 Reference Documents:\n",
            "\n",
            "Document 1:\n",
            "Node ID: 3ad27f99-efee-4049-8589-6b180a9eeb3b\n",
            "Text: Employee Key Contact Information.md 2025-07-10 4 / 6Employee\n",
            "IDNameSupervisorOffice LocationPosition TitlePhone NumberEmail Address\n",
            "Key Responsibilities EID-301Daniel KimSarah Johnson789 Market Ave\n",
            "#301Lead Textbook Editor(555) 345- 6789daniel.k@educompany.comContent\n",
            "curation, editorial standards EID-302Sophia PatelSarah Johnson789\n",
            "Market Ave #3...\n",
            "Score:  0.207\n",
            "\n",
            "\n",
            "Document 2:\n",
            "Node ID: 82511972-9e3f-4101-bf3f-0b4b23aa4288\n",
            "Text: Employee Key Contact Information.md 2025-07-10 3 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-210Jennifer LeeMichael Chen456 Tech\n",
            "Hub #210Data Analyst(555) 234-\n",
            "5687jennifer.l@educompany.comPerformance metrics tracking\n",
            "EID-211Christopher LeeMichael Chen456 Tech Hub #211Security Analy...\n",
            "Score:  0.203\n",
            "\n",
            "\n",
            "Document 3:\n",
            "Node ID: 56c11125-887a-4a0d-aefa-cbae9576759e\n",
            "Text: Employee Key Contact Information.md 2025-07-10 2 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-102Maria GonzalezSarah Johnson123\n",
            "Innovation Dr #102Senior Instructional Designer(555) 123-\n",
            "4568maria.g@educompany.comCourse optimization, learner engagement\n",
            "analysis EID-103James WilsonSara...\n",
            "Score:  0.196\n",
            "\n",
            "\n",
            "--------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "response = ask(\"Find Michael Johnson\", query_engine=query_engine)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 87,
      "id": "29964541",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8edb3d708c834e5cac63e8d67ae39e96",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Find Michael Johnson</td>\n",
              "      <td>There are two employees named Michael Johnson ...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Employee Key Contact Information.md 2025-07-1...</td>\n",
              "      <td>0.21101</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               question                                             answer  \\\n",
              "0  Find Michael Johnson  There are two employees named Michael Johnson ...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Employee Key Contact Information.md 2025-07-1...             0.21101   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display(evaluate_result(question, response, ground_truth))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0956716b",
      "metadata": {},
      "source": [
        "You can adjust the code to first retrieve 20 document slices from the vector database, then use the [text rerank](https://help.aliyun.com/zh/model-studio/getting-started/models#eafbfdceb7n03) provided by Alibaba Cloud Model Studio to re-rank them, and filter out the three most relevant reference pieces of information.\n",
        "\n",
        "After running the code, you will notice that, with the same three reference pieces of information, the LLM is now able to answer the question accurately.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 88,
      "id": "5b32f4f2",
      "metadata": {},
      "outputs": [],
      "source": [
        "from llama_index.postprocessor.dashscope_rerank import DashScopeRerank\n",
        "from llama_index.core.postprocessor import SimilarityPostprocessor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "id": "c59623e5",
      "metadata": {},
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine(\n",
        "    # First, set a larger number of recall slices\n",
        "    similarity_top_k=20,\n",
        "    streaming=True,\n",
        "    node_postprocessors=[\n",
        "        # In the rerank model, select the final number of slices you want to recall. Use the gte-rerank model from Tongyi Lab for reranking.\n",
        "        DashScopeRerank(top_n=3, model=\"gte-rerank\"),\n",
        "        # Set a similarity threshold; slices below this threshold will be filtered out\n",
        "        SimilarityPostprocessor(similarity_cutoff=0.2)\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "id": "26bdb676",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "==================================================\n",
            "🤔 Question: Which department is Michael Johnson in\n",
            "==================================================\n",
            "\n",
            "🤖 Answer:\n",
            "Michael Johnson appears in the Course Development Department and the Textbook Production Department. However, based on the provided information, there are multiple entries for Michael Johnson:\n",
            "\n",
            "1. **Course Development Department**\n",
            "   - **Position:** QA Specialist\n",
            "   - **Office Location:** 456 Tech Hub #205\n",
            "   - **Email:** michael.j@educompany.com\n",
            "\n",
            "2. **Course Development Department**\n",
            "   - **Position:** Technical Writer\n",
            "   - **Office Location:** 456 Tech Hub #209\n",
            "   - **Email:** michael.johnson@educompany.com\n",
            "\n",
            "3. **Course Development Department**\n",
            "   - **Position:** System Administrator\n",
            "   - **Office Location:** 456 Tech Hub #220\n",
            "   - **Email:** michael.j2@educompany.com\n",
            "\n",
            "If you need more specific information about which Michael Johnson you are referring to, please provide additional details or context.\n",
            "--------------------------------------------------\n",
            "📚 Reference Documents:\n",
            "\n",
            "Document 1:\n",
            "Node ID: 56c11125-887a-4a0d-aefa-cbae9576759e\n",
            "Text: Employee Key Contact Information.md 2025-07-10 2 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-102Maria GonzalezSarah Johnson123\n",
            "Innovation Dr #102Senior Instructional Designer(555) 123-\n",
            "4568maria.g@educompany.comCourse optimization, learner engagement\n",
            "analysis EID-103James WilsonSara...\n",
            "Score:  0.534\n",
            "\n",
            "\n",
            "Document 2:\n",
            "Node ID: 82511972-9e3f-4101-bf3f-0b4b23aa4288\n",
            "Text: Employee Key Contact Information.md 2025-07-10 3 / 6Employee\n",
            "IDName SupervisorOffice LocationPosition TitlePhone NumberEmail\n",
            "AddressKey Responsibilities EID-210Jennifer LeeMichael Chen456 Tech\n",
            "Hub #210Data Analyst(555) 234-\n",
            "5687jennifer.l@educompany.comPerformance metrics tracking\n",
            "EID-211Christopher LeeMichael Chen456 Tech Hub #211Security Analy...\n",
            "Score:  0.490\n",
            "\n",
            "\n",
            "Document 3:\n",
            "Node ID: 4fbb78ea-2a7b-415d-baa3-43ec53af6130\n",
            "Text: Employee Key Contact Information.md 2025-07-10 1 / 6Company\n",
            "Organizational Structure & Key Contact Information Corporate Mission\n",
            "Statement Our organization is dedicated to pioneering digital\n",
            "transformation in education through innovative learning solutions. As\n",
            "an edtech leader, we specialize in developing interactive online\n",
            "platforms that integr...\n",
            "Score:  0.317\n",
            "\n",
            "\n",
            "--------------------------------------------------\n"
          ]
        }
      ],
      "source": [
        "response = ask(\"Which department is Michael Johnson in\", query_engine=query_engine)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 79,
      "id": "636e00f2",
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1870972d2e1c4017be6405cccc8acebf",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Evaluating:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "      <th>ground_truth</th>\n",
              "      <th>contexts</th>\n",
              "      <th>answer_correctness</th>\n",
              "      <th>context_recall</th>\n",
              "      <th>context_precision</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Find Michael Johnson</td>\n",
              "      <td>Michael Johnson appears in the Course Developm...</td>\n",
              "      <td>There are three employees named Michael Johnso...</td>\n",
              "      <td>[Employee Key Contact Information.md 2025-07-1...</td>\n",
              "      <td>0.437653</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               question                                             answer  \\\n",
              "0  Find Michael Johnson  Michael Johnson appears in the Course Developm...   \n",
              "\n",
              "                                        ground_truth  \\\n",
              "0  There are three employees named Michael Johnso...   \n",
              "\n",
              "                                            contexts  answer_correctness  \\\n",
              "0  [Employee Key Contact Information.md 2025-07-1...            0.437653   \n",
              "\n",
              "   context_recall  context_precision  \n",
              "0            0.25                0.0  "
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "display(evaluate_result(question, response, ground_truth))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c282face",
      "metadata": {},
      "source": [
        "### 4.5 Answer generation phase\n",
        "\n",
        "Now, the LLM will generate the final answer based on your question and the retrieved content. However, this answer may still not meet your expectations. The issues you might encounter include:\n",
        "1. No relevant information was retrieved, and the LLM fabricated an answer.\n",
        "2. Relevant information was retrieved, but the LLM did not generate the answer as required.\n",
        "3. Relevant information was retrieved, and the LLM provided an answer, but you expected the AI to give a more comprehensive response.\n",
        "To address these issues, you can analyze and resolve them from the following perspectives:\n",
        "* Choosing the right LLM:\n",
        "    * For simple information queries and summaries, a small-parameter model is sufficient, such as [qwen-turbo](https://help.aliyun.com/zh/model-studio/models#ff492e2c10lub).\n",
        "    * If you want the Q&A bot to perform complex logical reasoning, it is recommended to choose a larger-parameter LLM with stronger reasoning capabilities, such as [qwen-plus](https://help.aliyun.com/zh/model-studio/models#bb0ffee88bwnk) or even [Qwen-Max](https://help.aliyun.com/zh/model-studio/models#cf6cc4aa2aokf).\n",
        "    * If your question requires reviewing a large number of document fragments, it is recommended to choose a model with a longer context length, such as [qwen-long](https://help.aliyun.com/zh/model-studio/models#27b2b3a15d5c6), [qwen-turbo](https://help.aliyun.com/zh/model-studio/models#ff492e2c10lub), or [qwen-plus](https://help.aliyun.com/zh/model-studio/models#bb0ffee88bwnk).\n",
        "    * If the RAG chatbot you are building is for non-general domains such as the legal field, it is recommended to use a model trained specifically for that domain, such as [Tongyi Farei](https://help.aliyun.com/zh/model-studio/models#f0436273ef1xm).\n",
        "* Fully optimize the prompt template. For example:\n",
        "    * Clearly request no fabrication of answers:\n",
        "        * LLMs may produce some inaccurate content, commonly referred to as hallucination.\n",
        "        * You can reduce the likelihood of LLM hallucinations by requiring in the prompt: \"If the provided information is insufficient to answer the question, please explicitly state 'Based on the available information, I cannot answer this question.' Do not fabricate answers.\"\n",
        "    * Add content delimiters: \n",
        "        * If the retrieved document slices are randomly mixed into the prompt, it will be difficult for humans to see the structure of the entire prompt, and the LLM will also be affected. \n",
        "        * It is recommended to clearly separate the prompt and the retrieved slices so that the LLM can correctly understand your intent.\n",
        "    * Adjust the template according to the type of question: \n",
        "        * Different types of questions may require different response paradigms. You can use the LLM to identify the question type and then map different prompt templates accordingly. \n",
        "        * For example, for some questions, you may want the LLM to first output the overall framework and then the details; for other questions, you may prefer the LLM to provide concise conclusions.\n",
        "* Adjust the parameters of the LLM. For example:\n",
        "    * If you want the LLM to produce the same output for the same question, pass the same seed value each time the model is invoked.\n",
        "    * If you want the LLM to avoid always using repetitive sentences when answering user questions, appropriately increase the presence_penalty value.\n",
        "    * If you are querying factual content, appropriately decrease the temperature or top_p values; conversely, when querying creative content,  increase their values.\n",
        "    * If you need to limit the word count (such as generating summaries or keywords), control costs, or reduce response time, appropriately lower the max_tokens value. However, if max_tokens is too low, it may lead to truncated output. Conversely, when generating long text, increase its value.\n",
        "    * Refer to the [Qwen API Reference](https://help.aliyun.com/zh/model-studio/use-qwen-by-calling-api) to learn more about the usage instructions for various parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "id": "96c18b00",
      "metadata": {},
      "outputs": [],
      "source": [
        "from llama_index.llms.openai_like import OpenAILike\n",
        "from llama_index.core import Settings\n",
        "import os"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "id": "8c98514a",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Factual query scenario - Low temperature, high certainty\n",
        "factual_llm = OpenAILike(\n",
        "    model=\"qwen-plus-0919\",  # Use the Qwen-Plus model\n",
        "    api_base=\"https://dashscope.aliyuncs.com/compatible-mode/v1\",\n",
        "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),\n",
        "    is_chat_model=True,\n",
        "    temperature=0.1,      # Lower temperature for more deterministic output\n",
        "    max_tokens=512,       # Control output length; however, if max_tokens is too small, it may lead to truncated output\n",
        "    presence_penalty=0.0, # Default presence_penalty\n",
        "    seed=42              # Fixed seed for reproducible output\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "id": "242a5ded",
      "metadata": {},
      "outputs": [],
      "source": [
        "# Creative scenario - High temperature, more diversity\n",
        "creative_llm = OpenAILike(\n",
        "    model=\"qwen-plus-0919\",\n",
        "    api_base=\"https://dashscope.aliyuncs.com/compatible-mode/v1\",\n",
        "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),\n",
        "    is_chat_model=True,\n",
        "    temperature=0.7,      # Increase temperature to make the output more creative\n",
        "    max_tokens=1024,      # Allow longer output\n",
        "    presence_penalty=0.6  # Increase presence_penalty to reduce repetition\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c085ee76",
      "metadata": {},
      "source": [
        "* Model fine-tuning for LLMs: If all the above methods have been thoroughly attempted but still fall short of your expectations, or if you hope to achieve further performance improvements, you can also try model fine-tuning tailored to your specific scenario. In later chapters, you will learn and practice this process."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "30451423",
      "metadata": {},
      "source": [
        "## ✅ Summary\n",
        "\n",
        "Through this section, you have gained an understanding the workflow of a simple RAG and common optimization techniques. You can also combine the knowledge you've acquired with your specific needs to route certain questions to different RAG chatbots, thereby building a more powerful modular RAG system. Additionally, from the previous lessons, you should also recognize that LLMs are not only useful for building question-answering systems: Leveraging LLMs to identify user intent and extract structured information—such as extracting tags from user questions as mentioned earlier—can also be applied in many other application scenarios.\n",
        "\n",
        "Of course, the optimization methods for RAG go far beyond those introduced in this course. The industry continues to research and explore RAG, and there are still many advanced RAG topics worth  studying. As shown through the previous learning, building a well-rounded and high-performing RAG chatbot is no simple task. In real-world applications, you may need to act quickly and won’t always have time to dive into every detail. Below are some directions worth exploring:\n",
        "\n",
        "* GraphRAG technology ingeniously combines the strengths of RAG and query-focused summarization (QFS), providing a powerful solution for handling large-scale text data. It integrates the advantages of both technologies: RAG excels at finding precise detailed information, while QFS is better at understanding and summarizing the overall content of an article. This combination allows GraphRAG to accurately answer specific questions and handle complex queries that require deeper understanding, making it particularly suitable for building intelligent question-answering systems. If you want to learn how to  apply GraphRAG in practice, you can refer to the detailed tutorial provided by LlamaIndex: [Building a GraphRAG Application with LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/cookbooks/GraphRAG_v2/).\n",
        "* With Model Studio, you can follow the document [Build a Private Knowledge Question-Answering Application Without Coding](https://help.aliyun.com/zh/model-studio/getting-started/build-knowledge-base-qa-assistant-without-coding) to quickly build a fairly effective RAG chatbot.\n",
        "* If your business processes are more complex, you can also leverage Visual Workflow,an agent orchestration application on Model Studio to build a more powerful application.\n",
        "* Model Studio also offers a range of [LlamaIndex components](https://help.aliyun.com/zh/model-studio/developer-reference/llamaindex/), allowing you to fully leverage Model Studio's capabilities while continuing to use the familiar LlamaIndex API to build RAG chatbots."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f132f563",
      "metadata": {},
      "source": [
        "## 🔥 Quiz\n",
        "\n",
        "### 🔍 Single choice question\n",
        "\n",
        "<details>\n",
        "<summary style=\"cursor: pointer; padding: 12px; border: 1px solid #dee2e6; border-radius: 6px;\">\n",
        "<b>In RAG applications, the length and content of document chunks significantly impact retrieval performance. If the chunk size is too large, introducing excessive noise, how should it be addressed❓(Select 1.)</b>\n",
        "\n",
        "- A. Increase the number of documents\n",
        "- B. Reduce the chunk size, or develop a more reasonable chunking strategy based on business characteristics\n",
        "- C. Use a more advanced retrieval algorithm\n",
        "- D. Improve the training level of the large model\n",
        "\n",
        "**[Click to view the answer]**\n",
        "</summary>\n",
        "\n",
        "<div style=\"margin-top: 10px; padding: 15px; border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
        "\n",
        "✅ **Reference Answer: B**  \n",
        "📝 **Explanation**: \n",
        "- Excessively long document chunks may include too much irrelevant information (noise), directly affecting retrieval accuracy.\n",
        "- For example, if a single chunk contains multiple topics, unrelated content may be retrieved during searches.\n",
        "- Optimizing the chunking strategy is the fundamental solution to address noise, as it controls input quality rather than relying on subsequent algorithmic or model compensation.\n",
        "\n",
        "</div>\n",
        "</details>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "1d2938e0",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "jupytext": {
      "cell_metadata_filter": "-all",
      "main_language": "python",
      "notebook_metadata_filter": "-all"
    },
    "kernelspec": {
      "display_name": "llm_learn",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
